title: Tellaro Query Language (TQL) - Rust class: repo-spec audience: in-repo status: current owner: tql-team last_verified: '2026-09-04' verification_note: 'CONTRADICTED on SEMANTICS again, and this page is the one that matters most for it: crates.io is what the Tellaro agent''s detection engine ships from, so a Rust-only defect changes the answer a saved rule gives on an endpoint while the backend query API stays correct and silent. The 2026-09-04 pass re-ran 102 literals through the Rust CLI and found 0 failures -- a PARSING result, which is exactly why it did not catch three Rust-only semantics changes that leave every one of those literals parsing. Added here, none of which had a doc surface anywhere in the repo (grep for "transpos|bareword|reversed" across every .md returned nothing): (12) Rust used to read f in [true] as a REVERSED comparison -- true eq ''f'', emitting {"match_phrase": {"true": "f"}}, zero hits and NO ERROR -- for every bracketed list whose elements all lex as barewords, including the multi-element [true, false]; and (13) Rust parsed value mutators and then never applied them, so f eq ''%41'' | urldecode matched {"f":"%41"} here and {"f":"A"} in Python. Also corrected: in/in_cs going element-wise was filed under the heading "Also user-visible, NOT BREAKING" -- it is breaking in Rust in the direction that ADDS records (f in [5] now matches a stored "05") and is now section 8; the matches/regexp non-string-operand half of the scalar-operand fix was absent -- now section 9; and the operator-less field | <predicate> form, where Rust was the MORE broken engine (all five IP predicates matched every record with the field, vs two of five in Python) -- now section 10. The intro said "six operator families ... and refuses a seventh" above SEVEN sections; it now defines the count as the section count, and the counts in this file, README.md, CHANGELOG.md and docs/TROUBLESHOOTING.md are derived from the headings by tests/unit/test_breaking_change_counts.py rather than typed. NO COUNT AND NO SECTION NUMBER IS RESTATED IN THIS NOTE, deliberately: this note twice carried a typed total that the next section addition falsified, and a section number typed here does not move when the sections are renumbered. A further false claim was fixed: the Type Hints parity warning said Rust and Python write a hint and a mutator chain in opposite orders and neither accepts the other''s -- closed by cea6024 on this branch. Executed against the Rust CLI built from this tree AND the Python engine at HEAD, not read: f in [true], f in [g] and f in [true,false] (forward reading, both engines agree); the in/eq matrix over [{f:5},{f:"5"},{f:"05"},{f:5.0}]; f eq ''%41'' | urldecode, its not contains twin, and the elementwise list form f eq [''%41''] | urldecode (matches {"f":["A"]} and NOT {"f":"A"}, in both); the value-mutator failure disposition including the zero-record case, which returns empty rather than raising; ip | is_loopback / ip | is_private against the ip | lowercase projection control; both hint/mutator orders on three shapes; and ANY tags = ''premium'' (still Rust-only, still correctly flagged). All 107 TQL-shaped literals in fenced blocks on this page and RUST_CLI_GUIDE.md were re-run through the Rust CLI: 107 parse, 0 fail. Three GeoIP examples were changed from geo() / geoip_lookup() to the bare names, which parse in BOTH runtimes -- the parenthesised spellings run only here, and they were the page''s only non-portable examples that carried no warning. Round 3 of the doc audit added the non-list in operand refusal: compare_in/compare_in_ci answered a constant false for a non-list operand and not_in is its negation, so f not in ''x'' matched EVERY record including the one holding exactly "x" -- a Rust-only fail-OPEN reachable from ordinary query text, and the only refusal of its class with no section. Both states were measured by running this crate''s CLI over [{f:"y"},{f:"x"}] with comparator.rs checked out at ffb52a7^ and again at HEAD, not read off the diff; f between ''x'' was run for the message this refusal is modelled on. Round 4 added a SHARED section for the top/bottom count Python refused inconsistently (ef65d17). This crate was already correct and is unchanged by it, but the section is shared rather than Rust-only because the release makes both runtimes refuse the same counts, so the whole Rust-only tail renumbered up by one -- which is why no section number is typed in this note. Rust''s side was re-verified by EXECUTION at HEAD, not read: this crate''s CLI refuses both stats count() by g top -1 and stats count() by g top 99999999999999999999 at parse, and accepts top 0, which is what makes "both engines agree on top 0" checkable. Not checked here: the stats/aggregation surface, the cache examples, and the performance figures.' verified_by: tql-docs-parity-campaign@2026-09-04 source_refs:

tellaro-query-language:tql/Cargo.toml
tellaro-query-language:tql/src
tellaro-query-language:tql/src/lib.rs
tellaro-query-language:tql/src/error.rs
tellaro-query-language:tql/src/stats_evaluator.rs
tellaro-query-language:tql/src/parser/grammar.pest
tellaro-query-language:tql/src/opensearch/mod.rs
tellaro-query-language:tql/src/field_accessor.rs
tellaro-query-language:tql/src/opensearch/query_builder.rs
tellaro-query-language:tql/src/opensearch/field_mappings.rs

Tellaro Query Language (TQL) - Rust

A blazing-fast, human-friendly query language for searching and filtering structured data in Rust.

TQL provides an intuitive SQL-like syntax for querying JSON, JSONL, CSV files, and OpenSearch indices with:

300x faster than Python for large file processing
First-class file support with CLI and programmatic API
OpenSearch integration with automatic DSL translation
25+ field mutators for data transformation (string, encoding, DNS, GeoIP, network)
Statistical aggregations for data analysis

use tellaro_query_language::Tql;
use serde_json::json;

let tql = Tql::new();
let records = vec![
    json!({"name": "Alice", "age": 30, "city": "NYC"}),
    json!({"name": "Bob", "age": 25, "city": "LA"}),
];

// Simple query
let results = tql.query(&records, "age > 27").unwrap();
assert_eq!(results.len(), 1);

// With field mutators
let results = tql.query(&records, "name | lowercase = 'alice'").unwrap();

🚀 Quick Start

Installation

Add this to your Cargo.toml:

[dependencies]
tellaro-query-language = "1.3"
serde_json = "1.0"

# Optional: Enable OpenSearch backend
# tellaro-query-language = { version = "1.3", features = ["opensearch"] }

CLI Installation

Install the high-performance command-line tool:

cargo install tellaro-query-language

# Query files directly
tql 'status = "active"' users.json
tql 'age > 25 AND city = "NYC"' data.jsonl

# Statistical aggregations
tql '| stats count() by status' events.jsonl
tql 'status = 200 | stats avg(response_time) by endpoint' logs.jsonl

⚠️ Upgrading: breaking changes in this release

This release makes eighteen breaking changes to what a query means in the Rust runtime: eleven applied identically in the Rust, Python and OpenSearch backends, and seven where Rust alone was wrong and converges on Python. Each is deliberate, and each is motivated by the same failure: a query that returned zero hits with no error, which in a detection rule is indistinguishable from "nothing happened".

Read this before upgrading a saved query, a detection rule, or anything that builds TQL from a template. This crate is what the Tellaro agent's detection engine runs, so sections 12–18 change the answer a saved rule gives on the endpoint even though the backend query API never had those defects.

The count is the number of numbered sections below — nothing else, and it is now asserted rather than asked for. It said "six operator families" while seven sections followed it; the recount that fixed that added this very paragraph as a guard, and the next commit added ### 14. without recounting anyway. Prose does not hold this. tests/unit/test_breaking_change_counts.py derives every number on this page — and in README.md, CHANGELOG.md and docs/TROUBLESHOOTING.md — from the headings themselves, so adding a section without recounting is now a test failure rather than a reading error.

1. `is null` and `not exists` now mean HAS NO VALUE

Both now mean absent or explicitly null, making them the exact complement of is not null / exists. The in-memory evaluator previously implemented a three-state model in which is null meant only "present and null" and not exists meant only "absent key" — so a record whose field was absent satisfied neither f is null nor f is not null.

No backend could ever have honoured that. OpenSearch does not index JSON nulls, so must_not: {exists: {field}} — which both DSL backends already emitted for both spellings — cannot tell an absent field from a null one.

What to check: saved queries using is null, is not null or not exists over a field that is sometimes absent. exists is unchanged. If you specifically wanted "present and explicitly null", write f eq null. Full matrix under Null and existence semantics.

2. `matches` / `regexp` searches unless you anchor it

f matches 'abc' and f matches '^abc$' emitted identical DSL, and both meant "the whole field value is exactly abc" — while the in-memory evaluator used a search and matched abc anywhere. The same query meant two things depending on where it ran. An unanchored pattern is now wrapped in .*; ^ and $ are still translated rather than wrapped.

Of 188 regex patterns in the shipped detection corpus, 5 were explicitly anchored and 151 were bare search patterns living on command-line, URL and path fields — where a full-value match is never what was meant.

What to check: OpenSearch queries with an unanchored pattern; they now match strictly more documents. If you relied on the implicit anchoring, write ^…$.

3. A LIST operand to `any` / `none` is refused

tags any ['a','b'] and every spelling of it now raise TqlError::TypeError — on translation and in the in-memory evaluator. The scalar form is unchanged, and a one-element list still unwraps to it. The clause previously produced either an HTTP 400 that failed the whole search, or a silent match_none — which made tags none [...] an exclusion clause that excluded nothing.

The evaluator was raised to the translator in a second step, after the two were measured selecting different record sets rather than different kinds of the same answer: a translator refusal is not local to the clause, it fails the whole query, so code eq 1 OR tags any ['a','b'] returned rows in memory and nothing through OpenSearch, and tags none ['a','b'] — an exclusion — returned every record in memory and nothing through OpenSearch.

What to write instead: tags in ['a','b']. The error names it. all / not_all are exempt in both layers.

4. Case-sensitive operators refuse a field that cannot answer them

contains_cs, startswith_cs, endswith_cs, in_cs and their not_ forms now raise TqlError::TypeError against an analyzed text field with no case-preserving subfield, instead of silently matching nothing. The analyzer lowercased every token it indexed, so the case being asked about is not in the index at all.

Measured on OpenSearch 2.19.4, analyzed field holding "A MiXeD Value":

wildcard *MiXeD*                 -> 0 hits
must_not(wildcard *MiXeD*)       -> EVERY document
wildcard *mixed*   (control)     -> EVERY document

The negated direction is the dangerous one: an exclusion that excludes nothing.

What to write instead: add a .keyword subfield to the mapping, or use the case-insensitive operator. The error names both. A .keyword or wildcard subfield, a keyword base field, non-string types (long, ip, date, boolean) and an unmapped field all still translate.

5. A type hint that cannot read a value skips the record, not the query

(Migration note reproduced verbatim from the commit that made the change.)

A query that previously ERRORED on a heterogeneous field now returns results. Nothing that previously returned results changes -- with one exception, and it is the one to check: a query using a NEGATED comparator with a type hint over a field whose values do not all read as the hinted type. Those records used to abort the query; they now answer false rather than being swept in. If any saved query relied on the abort as a data-quality alarm, that alarm is gone.

An author who wanted the strict behaviour should assert the shape explicitly rather than leaning on the hint's failure mode -- f::int exists and f::int > 75 selects records readable as an int, and a separate rule over the un-hinted field finds the ones that are not. There is no flag to restore the raise, deliberately: two evaluators with two modes is how the engines drifted before.

In Rust this lands on top of a larger change: type hints were inert entirely before this release. See Type Hints for the boundaries and for the four hint names Rust gained.

6. `::string` renders a boolean lowercase

flag::string eq 'true' over {"flag": true} previously matched nothing, because the hint rendered the boolean as True. It now matches; flag::string eq 'True' no longer does. JSON, TQL's own boolean literals and OpenSearch all write true/false; the capitalised spelling was the only one in the stack. Scope: a top-level boolean under string/str only.

7. The string operators render a boolean lowercase too

The string comparison operators do the same. f contains_cs true, startswith_cs, endswith_cs, matches / regexp and their not_ forms render a boolean as true / false on both sides — the operand written in the query and the value read out of the record. The evaluator and the OpenSearch translator had been selecting opposite documents for the same query: over [{"f": "x true y"}, {"f": "x True y"}] the evaluator returned the True record and wildcard *true* returned the true one, with no error on either side. OpenSearch settles it — term on a boolean field rejects "True" (only [true] or [false] are allowed). The case-insensitive forms lowercase both sides and are unaffected.

8. `in` / `in_cs` are element-wise `eq_ci` / `eq`

f in [a, b] is f eq_ci a or f eq_ci b, and f in_cs is the eq twin. Both engines re-implemented equality inside the membership arm instead of delegating to it, and drifted from it in opposite directions. They now delegate. in / not_in remain case-insensitive and in_cs / not_in_cs case-sensitive; that part is unchanged.

Measured over [{"f": 5}, {"f": "5"}, {"f": "05"}, {"f": 5.0}]:

Query	Before	Now (both engines)
`f in ['5']`	Python also matched `"05"`	`5`, `"5"`, `5.0`
`f in [5]`	Rust did not match `"05"`	`5`, `"5"`, `"05"`, `5.0`

The Rust half is the second row: compare_in_ci pre-lowercased every element with ast_value_to_string and compared a string field against that text, so a non-string element was answered as a string. f in [5] was false on the stored "05" while f eq_ci 5 — the same question with one element — was true, and f in_cs [5] was true as well. Rust disagreed with itself twice.

What to check: any membership query whose list holds a numeric-looking value, quoted or unquoted, over a field that stores numbers as text. Quoting is the author's type declaration; write [5] for the numeric reading and ['5'] for the text one.

The OpenSearch half is inert. A terms clause is resolved by the field's mapping, not the literal's JSON type — on OpenSearch 2.19.4 a long field holding 5 is selected by ["5"], [5], ["05"] and ["5.0"] alike. Only the in-memory evaluators, which have no mapping to consult, could select different records.

9. `matches` / `regexp` translate a non-string operand into a pattern

f matches true and f regexp 5 passed the operand straight through as raw JSON — {"regexp": {"f": true}} — Rust with the stated reasoning that "OpenSearch would report the type error". It does not. Measured on OpenSearch 2.19.4:

{"regexp": {"f": true}}        -> 0 hits, NO ERROR
{"regexp": {"f": ".*true.*"}}  -> 1 hit

The boolean is coerced to "true" and Lucene's regexp engine anchors implicitly, so the pattern must equal the whole value — a silent zero. {"regexp": {"f": null}} is worse: a hard HTTP 400 (value cannot be null) that fails the entire search, including every unrelated clause.

Both engines now render a non-string operand as a translated pattern. What to check: any matches / regexp clause with an unquoted true, false or number as the operand. This is one visible symptom of a wider fix spanning contains, startswith, endswith, all three _cs twins, matches, regexp, cidr, any and all, in both the positive and the negated spelling. The combination count this sentence used to give ("35 of 216" over an "18-operator × 6-literal matrix") was removed in round 4 because it reproduces on no axis — 18 × 6 is 108, the operators named here are eleven, and the matrix the suite pins in tests/unit/test_scalar_operand_parity.py is 8 × 6. Read it off that test rather than from prose.

10. `field | <predicate>` with no operator is a FILTER, not `exists`

When the last mutator in a chain returns a boolean — is_private, is_global, is_loopback, is_multicast, is_link_local — the operator-less form now means eq true. It used to parse to exists, so ip | is_loopback matched every record that merely had an ip field while reading as a filter. An unconditional-true clause dressed as a predicate cannot fail loudly; it quietly stops filtering and keeps returning plausible results.

Rust was the more broken of the two engines here, not the less: parse_field_only_expression enumerated nothing and emitted exists for every mutator, so all five predicates matched everything. (Python listed two of the five in three hand-written places, so two filtered and three did not.) Rust now derives the set from a Mutator::returns_boolean method rather than naming any predicate in the parser, and a test pins its membership equal to Python's.

Reverting tql/src/parser/mod.rs makes ip | is_loopback return ["127.0.0.1", "8.8.8.8", "10.0.0.1", "224.0.0.1", "169.254.1.1"] again — every record with an ip field.

A transforming mutator is unchanged. ip | lowercase is a projection — keep every record that has the field, apply the mutator on the way out — and still parses to exists.

What to check: any saved rule using one of the five IP predicates with no comparison operator. The explicit spellings (ip | is_loopback = true) are unchanged.

11. A `top` / `bottom` count the engine cannot honour is refused at every surface

This crate was already right, and is unchanged by it — the fix is Python's, and it is listed here because it closes the divergence section 16 opens from the other side. Rust refuses a count outside 0..=usize::MAX at parse; Python now refuses the same counts at validate(), evaluate(), to_opensearch() and query() alike, so the two runtimes finally answer one saved query the same way.

Python refused overflow nowhere: stats count() by g top 99999999999999999999 passed validate() and emitted {"terms": {"field": "g", "size": 99999999999999999999}} — a size OpenSearch parses into a 32-bit int and rejects — while this crate refused the identical query when it was written. And Python refused a negative count only on the surfaces that CHECK a query: query() routes through neither validate() nor to_opensearch(), so stats count() by g top -1 over five groups returned four buckets with health_status: "green" and no error, results[:-1] silently dropping the last one.

top 0 is unaffected in both runtimes — it is representable, so this parser has nothing to observe about it, and Python still refuses it only in the translator.

What to check: nothing in a Rust consumer. If the same saved query also runs against the backend query API, it now raises there instead of answering one bucket short.

The next seven affect the Rust runtime only. Python and the TypeScript editor already behaved this way, so a query written against the backend query API never saw them — but the agent's detection engine runs this crate.

12. An empty list literal is refused at parse

f in [] parsed here and was refused by the other two engines — Python raised TQLSyntaxError and the TypeScript editor reported Empty value list. Python is what the backend query API runs and the editor is what autocompletes queries, so this was a query that was green while you typed it, refused when you saved it, and valid in the agent's detection engine.

The rule is shared by every list-taking operator, so in, not_in, in_cs, not_in_cs, any, all, none and between all refuse an empty literal now. A non-empty list is untouched, and f in [''] is still a list containing one empty string.

The original reason to refuse — both translators rendered an empty membership list as {"bool":{"should":[],"minimum_should_match":1}}, which OpenSearch reads as match_all, so f in [] matched every document and f not_in [] matched none — no longer applies: that shape was converged onto {"terms":{"f":[]}}, which does not invert. What settles it is that an empty membership list cannot express an intent. It is what a rule template emits when its value list came from a variable that happened to be empty.

13. `f in [bareword]` no longer transposes the field and the value

This one has no loud failure mode at all, and it is the reason to read this page before upgrading a detection rule. In the Rust parser, in_fields_list is a list of bare field_names and value_with_mutators accepts a bare identifier, so every list whose elements all lex as bare identifiers was a well-formed reversed-in expression. in_fields_comparison was listed ahead of binary_comparison in the comparison alternation, and PEG's ordered choice took that reading and never reached the forward one:

f in [true]   parsed as   {field: "true", operator: "eq", value: "f"}

That is a comparison against a field literally named true, and the DSL it emitted was {"match_phrase": {"true": "f"}} — zero hits and no error, indistinguishable from "nothing matched".

The affected spellings are every bracketed list whose elements are all barewords: f in [true], f in [false], f in [null], f in [g], f in [true, false]. It was not limited to single-element lists — f in [true, false] became true eq 'f' OR false eq 'f'. f in [true, 'a'] parsed correctly only by accident: one quoted element breaks the field-list reading, and that accident is what made the defect look narrower than it is.

Python has never had this — field_in_values is listed ahead of value_in_field_list there, so the forward reading always won. The semantics adopted are Python's, verbatim: the forward reading wins for a bracketed list, and the reversed reading is reachable only when the left operand cannot be read as a field name (a quoted string or a number). No query the backend query API accepts today changes meaning.

The reversed spellings that must survive, and do:

'val' in [f1, f2]     // bracketed: LOSES to the forward reading
'val' in arrayfield   // unbracketed: still means "arrayfield contains 'val'"

Those sit on opposite sides of binary_comparison, which is why the grammar gained a separate in_field_comparison rule instead of the one-line alternation reorder that would silently have flipped the second form.

What to check: every saved rule with a bracketed in list whose elements are unquoted words. It was returning nothing and reporting success.

14. Value mutators are applied, not parsed and ignored

f eq '%41' | urldecode matched {"f": "%41"} in Rust and {"f": "A"} in Python — same query, two answers, no error.

A value mutator transforms the literal the author typed, before comparison. The parser built ComparisonNode::value_mutators correctly all along; the evaluator never read the field, so the comparator always saw the raw literal.

Three semantics, established by running Python rather than reading it:

A list operand mutates elementwise. f eq ['%41'] | urldecode becomes ["A"], not the mutated text of the whole list — it matches {"f": ["A"]} and not {"f": "A"}.
Mutators chain left to right, as on the field side.
A failure is a QUERY error, not a data error. f eq '%FF' | urldecode raises TqlError::ValidationError (Python's TQLValidationError) rather than answering false, because the operand comes from the query text and so fails identically for every record. One measured wrinkle: the raise happens on the evaluation path, so a query over zero records returns empty and does not raise, in both engines. The field-mutator direction keeps its leniency — an unreadable record value skips that record and the scan carries on.

This also fixed every not <op> spelling in Rust. f not contains '%41' | urldecode returned a hit for {"f": "A"} and none for {"f": "%41"} — an exclusion clause excluding exactly the wrong record. It is now the other way round. Rust's parser was never wrong here: it already produced operator: "not_contains", value: "%41", value_mutators: [urldecode], so one evaluator fix closed all of them.

Still open, on the Python side only. Python's parser folds the mutator into the value for the negated spellings, so f not contains '%41' | urldecode still drops the mutator there and matches both records. Its strict xfail is deliberately left in place and flips green when Python's parser is fixed. If you need that spelling today, Rust is the engine that answers it correctly.

15. A boolean predicate no longer overwrites the field it filtered on

ip | is_loopback returned {"ip": true} in Rust and {"ip": "127.0.0.1"} in Python — same query, same MATCH SET, two different payloads.

A predicate answers a boolean about a value; it is not a request to replace the value with that boolean. filter_and_enrich wrote every mutator's output back over its field, so the address an analyst filtered for was destroyed by the filter that selected it. All five IP predicates were affected (is_private, is_global, is_loopback, is_multicast, is_link_local), in the bare, exists and eq true spellings, and the negated form projected false the same way. A nested field was worse: source.ip | is_loopback returned the record with its nested address intact and a flat "source.ip": true key grafted on beside it.

The counts always agreed, which is why nothing caught it for so long: every predicate matched exactly the right records in both engines, and the shared cross-language fixture format has no key for an expected returned record — only expect_match, a boolean per record. A count assertion is structurally incapable of seeing this defect.

Python fixed the identical shape on its own side earlier (TYPE_CHANGING_MUTATORS in src/tql/mutator_classification.py), where source.ip | is_loopback exists returned {"source": {"ip": true}} while is_private three lines away preserved the address. Rust converges onto that behaviour for every chain that ends in the predicate — the bare ip | is_loopback, the exists and eq true spellings, the negated form, the nested field, and ip | lowercase | is_loopback all now return the same payload in both engines. Measured 2026-09-04 by running both, not inferred.

One spelling has NOT converged, and it is this section's own defect wearing a different chain order. ip | is_loopback | lowercase — a transform written after the predicate — still returns {"ip": true} / {"ip": false} in Rust, while Python returns the address unchanged. Both engines select the same records (every record with the field: the last mutator is not a predicate, so the clause reads as exists), so the counts agree and only the payload differs — the exact shape that hid the original bug for so long.

The two engines ask different questions. Rust suppresses write-back when the last mutator in the chain returns a boolean, so any transform after a predicate re-enables projection. Python's post-processor never applies the predicate to the value it returns at all, so the address survives whatever follows it in the chain.

Which rule is correct is a semantics decision, not a documentation fix, so nothing here changes it. It is recorded under Known gaps in CHANGELOG.md and pinned in both suites (tql/tests/predicate_does_not_project.rs, tests/unit/test_predicate_does_not_project.py) so today's answer cannot change silently in either engine. Measured 2026-09-04 on ip | is_loopback | lowercase and ip | is_loopback | uppercase.

A projecting mutator is unchanged and must stay that way. name | lowercase replacing name with its lowercase form is the entire purpose of a projection, so the fix distinguishes the two classes rather than disabling write-back: it asks Mutator::returns_boolean — the same declaration the parser already uses to decide that field | <predicate> is a filter — about the last mutator in the chain. A sixth predicate overrides that one method and needs no edit here. ip | lowercase | is_loopback therefore preserves and name | lowercase projects, both verified. Asking only the last mutator is also why the reverse order, ip | is_loopback | lowercase, still projects here and is the residual divergence noted above.

What to check: any Rust consumer of Tql::query_enriched, Evaluator::filter_and_enrich, the file-query path, or the OpenSearch post-processor that reads a field an IP predicate was applied to. It was receiving true/false where it expected the address. Tql::query is unaffected — it returns borrowed records and never projected.

16. A `top` / `bottom` count this runtime cannot represent is refused, not replaced with 10

| stats sum(salary) top -1 by department parsed here as top 10. So did | stats count() by department top -1, on the group-by side.

The grammar's integer is "-"? ~ ASCII_DIGIT+, so a negative count is well-formed source. The parser then read it into an Option<usize> with .parse().unwrap_or(10) — a fallback written to supply a default for a missing count, firing instead for a malformed one. Absent and unrepresentable got the same answer. Measured 2026-09-04 before the fix, by running both engines:

Query	Rust parsed to	Same query through Python
`stats sum(salary) top -1 by department`	`modifier: Some("top"), limit: Some(10)`	refused: "asks for a non-positive number of buckets"
`stats count() by department top -1`	`bucket_size: Some(10)`	refused: "asks for a negative number of buckets"

So one saved query got ten buckets from the agent's detection engine and an error from the backend. The two answers were not merely different: one of them was invented. A live cluster rejects the DSL the Rust reading would have produced ([size] must be greater than 0), so there was no runtime at which the substituted 10 was the author's request.

A count larger than usize::MAX took the same path and produced the same top 10.

The message deliberately does not match Python's. Python's parser carries a signed integer all the way to its translator, so opensearch_stats.py can see a negative limit and call it non-positive. Option<usize> cannot hold a negative at all, so what this runtime observes is narrower — a leading -, or a count past usize::MAX. Borrowing Python's phrasing would describe a state this type excludes.

An absent count still defaults, and top 0 still parses. A modifier written with no number produces no integer pair, so limit / bucket_size stay None and every downstream default is unchanged. 0 is representable, so the parser has nothing to observe about it; it is refused later by the OpenSearch translator, exactly as before. Only a count that cannot be represented is refused here.

What to check: any saved query, detection rule or template that computes a top N where N can reach zero-or-below by arithmetic — top <count - 1> against an empty count, say. Under the agent it was silently answering with ten buckets; it now refuses at parse. A positive or absent count is unaffected.

17. A non-list operand to `in` is refused, not answered `false`

f not in 'x' matched every record, including the one holding exactly "x" — an exclusion that excluded nothing and reported success.

compare_in and compare_in_ci ended in _ => Ok(false) for an operand that was not a list, and compare computes not_in as !compare_in_ci(..)?. A constant false inverts to a constant true. Measured 2026-09-04 over [{"f":"y"},{"f":"x"}], by running this crate's CLI at the commit before the fix and at the commit after it:

Query	Before	After
`f in 'x'`	0 of 2 matched	`Operator error: IN operator requires a list of values`
`f not in 'x'`	2 of 2 matched — `y` and `x`	the same refusal
`f in_cs 'x'`	0 of 2 matched	the same refusal
`f not in_cs 'x'`	2 of 2 matched	the same refusal

The positive and negative spellings were not complements, and the negative one was the dangerous direction: a rule reading "exclude these values" admitted every record it was written to drop. grammar.pest's in_list admits the bare scalar, so this was reachable from ordinary query text rather than only from a hand-built AST.

OperatorError, and not ValueError, matching compare_between — the neighbouring operator that already refused its own non-list operand, and the reason this shape was findable at all. f between 'x' answers Operator error: BETWEEN operator requires a list of exactly 2 values.

This is the in-memory evaluator only. The change is confined to comparator.rs; the OpenSearch query builder has no arm for a scalar in operand and is untouched, so no translated DSL moves.

What to check: any saved rule or template that writes in, not in, in_cs or not in_cs against a single unbracketed value — most likely a template that renders one item into f not in '{{value}}'. Under this runtime the two negative spellings were matching everything; they now refuse at evaluation. The list form is unaffected and has always been correct: f in ['x'] selects {"f":"x"} and f not in ['x'] selects {"f":"y"}, in both runtimes.

Python is wrong differently here, and is deliberately not changed — it still answers a scalar operand, and answers inconsistently with itself. The measured matrix is in Known gaps in CHANGELOG.md; write the list and the divergence cannot reach you.

18. A presence predicate carrying a mutator returned the exact complement

f | md5 not_exists translated to {"exists": {"field": "f"}} — precisely the documents that do not match, returned as hits.

build_query_clause consulted has_post_processing_mutators at the very top, before the not_exists arm and before build_comparison could reach its is / is_not / eq null / ne null arms, and answered a bare exists clause for any comparison carrying one of the 24 non-collection mutators. For a real comparison that rewrite is the intended over-broad phase-1 net: fetch everything that has the field, let post-processing narrow it. A presence predicate is not approximate — its translation is exact — so the same rewrite does not over-broaden it, it inverts it.

Query	Before	After (and Python, always)
`f not_exists`	`{"bool":{"must_not":{"exists":{"field":"f"}}}}`	unchanged
`f \| md5 not_exists`	`{"exists":{"field":"f"}}`	`{"bool":{"must_not":{"exists":{"field":"f"}}}}`

Five of the ten presence spellings were affected, not one: not_exists, not exists, is null, eq null and = null. The other five — exists, is not null, is_not null, ne null, != null — were unharmed only by the coincidence that exists is already their correct answer, not because anything protected them. Measured across all 10 spellings × every entry of mutators::MUTATOR_NAMES × 2 mapping shapes: every combination naming a mutator that blocks pushdown inverted, and no other. The collection mutators — any, all, avg, average, sum, min, max — do not block pushdown, so the rewrite never fired for them and they were never affected. No count is written here on purpose: it is the product of two derived sets, and a figure typed into prose is what this release has already had go wrong.

The gate is now has_operand, a property of the node rather than a list of operator names. Absence needs two spellings here where Python needs one: a valueless operator (exists, not_exists) parses to value: None, while the null literal (is null, eq null) parses to Some(AstValue::Null). Matching only None fixes the valueless spellings — not exists and not_exists — and leaves is null / eq null / = null inverted, reporting itself green: which is why this is a named function and not an is_some() at the call site, and why the partial fix is pinned by mutation rather than by reasoning.

Python's polarity was correct throughout — it never emitted the complement — so the clause ordering in this crate was the entire divergence on that axis. It was documented in two Python docstrings as a known Rust defect deliberately not copied, and asserted in no Rust test at all. Python's field resolution was a separate defect and not correct: eq null and = null resolved against .keyword rather than the base field until a4f482d, which the Breaking — Python runtime and the OpenSearch backend only section of CHANGELOG.md covers.

What to check: any detection rule that asks whether a field is absent while also naming a mutator on it — user.name | lowercase not_exists, hash | md5 is null. Under this runtime it was selecting every record that had the field. Rules using the positive spellings (exists, is not null) were returning the right records throughout and need no review. The bare predicate with no mutator was never affected in either runtime.

Also user-visible, not breaking

cidr / not_cidr translate to OpenSearch. The Rust query builder had no arm for them, so both fell through to "Unsupported operator" while the parser and the in-memory evaluator accepted them — a shipped bundled rule using cidr could never execute on an agent. Both evaluators now also iterate array-valued fields: related.ip cidr '10.0.0.0/8' matches ["10.1.2.3", "8.8.8.8"]. cidr is refused on a keyword-mapped field, where term compares the literal string "10.0.0.0/8"; map the field as ip.
Word operators have boundary guards. f nonexists — the obvious typo for f not exists — parsed as none with the value 'xists' and returned a match. It is now a parse error, as are f containszq, f eqzq, f isnull and f::intzq.
Four mutator names are rejected at parse: count, unique, first, last.

📁 Query Files (First-Class Support)

CLI Usage

TQL treats files as first-class data sources:

# Query JSON/JSONL files
tql 'status = "active"' users.json
tql 'age > 25 AND city = "NYC"' data.jsonl

# Query CSV files (auto-detects headers)
tql 'price > 100 AND category = "electronics"' products.csv

# Statistical aggregations
tql '| stats count() by status' events.jsonl
tql 'status = 200 | stats average(response_time) by endpoint' logs.jsonl

# Process folders recursively
tql 'level = "ERROR"' logs/ --pattern "*.jsonl" --recursive

# Stream data from stdin
cat large-file.jsonl | tql 'score > 90'

# Output formats
tql 'age > 30' users.json --output results.json   # JSON
tql 'age > 30' users.json --output results.jsonl  # JSONL
tql 'age > 30' users.json                         # Table (console)

Performance: Process 50MB files in ~200ms with streaming (no memory overhead).

Programmatic File Queries

use tellaro_query_language::Tql;
use std::fs::File;
use std::io::BufReader;
use serde_json::Value;

let tql = Tql::new();

// Read and query JSON file
let file = File::open("data.json")?;
let reader = BufReader::new(file);
let records: Vec<Value> = serde_json::from_reader(reader)?;
let results = tql.query(&records, "status = 'active' AND age > 25")?;

// Stream JSONL for large files
let file = File::open("large.jsonl")?;
let reader = BufReader::new(file);
for line in reader.lines() {
    let record: Value = serde_json::from_str(&line?)?;
    if tql.matches(&record, "level = 'ERROR'")? {
        println!("Error found: {}", record);
    }
}

🗄️ OpenSearch Integration

TQL seamlessly integrates with OpenSearch/Elasticsearch:

Automatic DSL Translation

use tellaro_query_language::{Tql, opensearch::{OpenSearchClient, QueryBuilder}};

// Configure OpenSearch (reads from environment)
std::env::set_var("OPENSEARCH_HOSTS", "http://localhost:9200");
std::env::set_var("OPENSEARCH_USERNAME", "admin");
std::env::set_var("OPENSEARCH_PASSWORD", "admin");

// Create client
let config = OpenSearchConfig::from_env()?;
let client = OpenSearchClient::new(config)?;

// Parse TQL query
let tql = Tql::new();
let ast = tql.parse("age > 25 AND status = 'active'")?;

// Build OpenSearch DSL
let builder = QueryBuilder::new(None);
let opensearch_query = builder.build_query(&ast)?;

// Execute search
let response = client.client()
    .search(opensearch::SearchParts::Index(&["users"]))
    .body(opensearch_query)
    .send()
    .await?;

TQL → OpenSearch Query DSL

TQL automatically translates to optimized OpenSearch queries:

TQL Operator	OpenSearch Query	Example
`eq`, `=`	`term` or `match_phrase`	`status = "active"`
`ne`, `!=`	`bool` + `must_not`	`status != "deleted"`
`gt`, `gte`, `lt`, `lte`	`range`	`age > 25`
`contains`	`wildcard` + `case_insensitive`	`email contains "@example.com"`
`startswith`	`wildcard` + `case_insensitive`	`name startswith "John"`
`endswith`	`wildcard` + `case_insensitive`	`filename endswith ".pdf"`
`contains_cs`, `startswith_cs`, `endswith_cs`	`wildcard`, no `case_insensitive`	`name startswith_cs "John"`
`matches` (regexp)	`regexp`, *unanchored patterns wrapped in `.`**	`email matches "^\\w+@\\w+"`
`in`	`bool` + `should` of `term` + `case_insensitive`	`status in ["active", "pending"]`
`in_cs`	`terms` (case-sensitive)	`status in_cs ["Active"]`
`between`	`range` with `gte` + `lte`	`age between [18, 65]`
`cidr`	`term` on an `ip`-mapped field	`ip cidr "192.168.0.0/16"`
`not_cidr`	`bool` + `must_not` + `term`	`ip not cidr "10.0.0.0/8"`
`exists`, `is not null`	`exists` on the base field	`field exists`
`not exists`, `is null`	`bool` + `must_not` + `exists`	`field is null`
`AND`	`bool` + `must`	`age > 25 AND city = "NYC"`
`OR`	`bool` + `should`	`city = "NYC" OR city = "LA"`
`NOT`	`bool` + `must_not`	`NOT status = "deleted"`

Field Mapping Intelligence

use tellaro_query_language::opensearch::FieldMappings;

// Get mappings from OpenSearch
let mappings_response = client.client()
    .indices()
    .get_mapping()
    .index(&["users"])
    .send()
    .await?;

let mappings = FieldMappings::from_opensearch_response(
    mappings_response.json().await?
)?;

// Use mappings for intelligent query generation
let builder = QueryBuilder::new(Some(mappings));
let query = builder.build_query(&ast)?;
// Automatically selects .keyword for exact matches on text fields

📖 Syntax Guide

Comparison Operators

// Equality
"status = 'active'"           // Exact match (alias: eq)
"status != 'inactive'"        // Not equal (alias: ne)

// Numeric comparisons
"age > 25"                    // Greater than
"age >= 18"                   // Greater or equal
"age < 65"                    // Less than
"age <= 100"                  // Less or equal

// String operations (case-INSENSITIVE)
"email contains '@example.com'" // Substring match
"name startswith 'John'"      // Prefix match
"filename endswith '.pdf'"    // Suffix match

// Case-SENSITIVE twins, and their negations
"name contains_cs 'John'"     // also startswith_cs, endswith_cs
"name not_contains_cs 'John'" // also not_startswith_cs, not_endswith_cs

// Pattern matching. An UNANCHORED pattern SEARCHES; ^...$ means the whole value.
"email matches 'example\\.com'"       // matches anywhere in the value
"email matches '^\\w+@\\w+\\.\\w+$'"  // matches the whole value

// Range and membership (in / not in are case-INSENSITIVE)
"age between [18, 65]"        // Inclusive range
"status in ['active', 'pending']"     // Value in list
"status not in ['deleted', 'archived']" // Value not in list
"status in_cs ['Active']"     // case-sensitive; also not_in_cs

// IP operations
"ip cidr '192.168.0.0/16'"    // IP in CIDR range

// Existence checks. `exists` and `is not null` are the SAME question,
// and so are `not exists` and `is null`.
"field exists"                // Field is present AND non-null
"field is not null"           // identical to `field exists`
"field not exists"            // Field is absent OR present-and-null
"field is null"               // identical to `field not exists`

Null and existence semantics

The two pairs are exact complements — every record satisfies exactly one side:

document	`is null`	`not exists`	`is not null`	`exists`
`{}`	match	match	--	--
`{"f": null}`	match	match	--	--
`{"f": "x"}`	--	--	match	match
`{"f": []}`	--	--	match	match

An empty array is a present value: the producer deliberately wrote a field with zero elements, so [] satisfies exists and is not null. [null] and ["a", null] follow the same rule.

Residual divergence, stated rather than left to be found. OpenSearch indexes neither a JSON null nor an empty array, so against a cluster {"f": []} and {"f": [null]} DO match not exists / is null. The in-memory evaluator reads the record itself and answers per the table above. This is the one document shape where the two execution paths differ.

Regex semantics

matches and regexp are the same operator. An unanchored pattern searches — f matches 'abc' matches any value containing abc. Against OpenSearch the pattern is wrapped in .*, because Lucene's regexp query anchors implicitly; a pattern carrying ^ or $ is translated unwrapped, so an author who anchored deliberately keeps full-match semantics.

Logical Operators

// AND (all conditions must be true)
"age > 25 AND city = 'NYC'"
"status = 'active' AND role in ['admin', 'moderator']"

// OR (either condition must be true)
"city = 'NYC' OR city = 'LA'"
"status = 'admin' OR role = 'superuser'"

// NOT (negates condition)
"NOT (age < 18)"
"NOT status = 'deleted'"

// Complex expressions with parentheses
"(age > 25 AND city = 'NYC') OR (status = 'vip' AND score > 90)"

Collection Operators

// ANY - at least one array element matches
"ANY tags = 'premium'"
"ANY user.roles = 'admin'"

// ALL - every array element matches
"ALL scores >= 80"
"ALL status = 'active'"

// NONE - no array elements match
"NONE flags = 'spam'"
"NONE violations.severity = 'critical'"

Parity warning. These operator-first forms are Rust-only. The Python grammar accepts only the field-first, operator-less form (tags any 'premium') and raises TQLSyntaxError on ANY tags = 'premium'. Use the field-first form for queries that must run under both.

Stats top-N is now portable in both spellings. Both engines accept the modifier inside the parens (| stats sum(x, top 10) by y) and after them (| stats sum(x) top 10 by y), and both build the same modifier + limit. Until 2026-09-04 each engine accepted only its own spelling and rejected the other's, so there was no portable spelling at all -- and the warning that stood here stated only the Rust half, which implied an alternative that did not exist.

Both engines now also MEAN the same thing by it. Until 2026-09-04 the gap after the grammars were reconciled was behavioural and it was in Rust: tql/src/stats_evaluator.rs consumed neither modifier, so | stats count() by role top 3 answered 9 buckets where Python answers 3, and | stats sum(salary) top 3 by department answered 13 where Python answers 3 -- the same 13 the un-modified query returns. A limit that is parsed and then dropped answers with more rows than were asked for, which reads as a complete result rather than as an error.

The two modifiers are not one modifier, and this is the part worth remembering:

spelling AST field ranks buckets by

sum(x) top 3 by y Aggregation.modifier + .limit the aggregate value

sum(x, top 3) by y Aggregation.modifier + .limit the aggregate value

... by y top 3 GroupBy.bucket_size doc_count

A query carrying both applies the aggregation modifier first and the bucket limit second, so the survivors are re-ranked; swapping the passes yields a different set, not merely a different order. Bucket emission order is first-appearance (record order) in both engines, and both top-N passes sort stably, so emission order is what breaks ties.

Two Python quirks are reproduced in Rust deliberately rather than corrected, because two engines disagreeing is worse than one odd answer: top 0 on a single group-by field is a complete no-op while on a multi-level grouping it empties the result, and only the first aggregation carrying a modifier is applied. One divergence is deliberate: a bucket whose aggregate is null sorts as zero in Rust, where Python raises TypeError -- a divergence from an exception, not from a semantics.

Against a CLUSTER, the answer is the same -- with three stated exceptions. Until 2026-09-04 the translators pushed neither modifier down: the aggregation limit was computed and discarded (# noqa: F841 in opensearch_stats.py, never read at all in stats_translator.rs) and an absent bucket size was defaulted to five, so | stats sum(x) top 3 by y asked the cluster for five buckets and | stats sum(x) by y asked for five where memory returns every one. Both now emit size and order per modifier, with a bucket_sort sub-aggregation as the second pass when a query carries both.

Three things a terms aggregation cannot be made to do, measured against OpenSearch 2.19.4 and pinned by tql/tests/stats_top_n_pushdown_live.rs and tests/integration/test_stats_top_n_pushdown_opensearch.py:

Tie-breaks. terms breaks a tie on its order key with _key ascending; both in-memory engines break it in first-appearance order, which no terms order expresses. | stats sum(salary) by department top 5 over the shared corpus cuts inside a four-way doc_count = 2 tie, so the fifth bucket is Finance in memory and Data Science from a cluster.

Un-modified bucket order. The set agrees; terms returns _count descending where memory returns first-appearance order.

Multi-level bucket limits. Nested terms picks each level's top N by that level's own doc_count; the in-memory pass sorts the flattened key combinations and reserves a slot per parent key. Different set and different count, not merely a different order.

Four spellings are refused at translation time rather than answered differently, because the DSL that would express them is one OpenSearch rejects: an aggregation modifier alongside more than one group-by field, top 0 on an aggregation, an aggregation modifier ranking by a multi-value metric (median, percentile, std, percentile_rank) or by the listing family (values/unique/distinct), and top 0 on a group-by field in a multi-level grouping. All four remain answerable in memory; the refusal is about this backend, not about the language.

spelling	AST field	ranks buckets by
`sum(x) top 3 by y`	`Aggregation.modifier` + `.limit`	the aggregate value
`sum(x, top 3) by y`	`Aggregation.modifier` + `.limit`	the aggregate value
`... by y top 3`	`GroupBy.bucket_size`	`doc_count`

The operand of any / none is a scalar. tags any ['a','b'] does not ask "is any element of tags one of these two values" — it asks whether a single element equals the two-element list, which no scalar element can. Both the OpenSearch translator and the in-memory evaluator raise TqlError::TypeError on it and name in:

"tags any ['a', 'b']"     // TqlError::TypeError -- ill-typed operand
"tags in ['a', 'b']"      // what that query meant
"tags any 'a'"            // fine; a one-element list unwraps to this

all / not_all are the deliberate exception: they are answered by a Painless script that compares each element to the operand, where a list is a valid (never-matching) value, in both layers.

The refusal is decided from the query alone, never from the data: and / or short-circuit, so a check made only where the clause is reached would refuse code eq 1 OR tags any ['a','b'] for a corpus containing a code != 1 record and quietly answer it for one that does not. A query that is well-typed or not depending on the records is the silent shape this refusal exists to remove.

Nested Field Access

// Dot notation for nested objects
"user.profile.email contains '@example.com'"
"metadata.tags.priority = 'high'"

// Array indexing uses dot notation -- bracket syntax is NOT accepted
"tags.0 = 'urgent'"
"history.5.status = 'completed'"

🔄 Field Mutators (25+)

Transform field values inline before comparison:

String Mutators

// Case conversion
"email | lowercase contains '@example.com'"
"name | uppercase = 'JOHN DOE'"

// Whitespace handling
"message | trim = 'hello'"

// String manipulation
"text | length > 100"
"path | split('/') | length = 3"
"text | replace('old', 'new') contains 'new'"

Encoding Mutators

// Base64
"data | b64encode = 'aGVsbG8='"
"encoded | b64decode contains 'secret'"

// URL encoding
"param | urldecode = 'hello world'"

// Hexadecimal encoding
"data | hexencode = '68656c6c6f'"
"encoded | hexdecode = 'hello'"

// Cryptographic hashing
"password | md5 = '5f4dcc3b5aa765d61d8327deb882cf99'"
"data | sha256 = '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'"

Network/Security Mutators

// Defang/Refang URLs (security analysis)
"url | defang contains 'hxxp://example[.]com'"
"indicator | refang = 'http://malicious.com'"

// IP address classification
"source_ip | is_private = true"      // Check if IP is RFC 1918 private
"dest_ip | is_global = true"         // Check if IP is globally routable

// Use cases
"source_ip | is_private = true and port = 22"  // Internal SSH connections
"dest_ip | is_global = false"                    // Non-routable destinations

DNS Mutators

// DNS lookups with caching
"hostname | nslookup contains '8.8.8.8'"
"domain | nslookup = '1.1.1.1'"

Performance: DNS results are cached in memory to avoid repeated lookups.

GeoIP Mutators

// GeoIP enrichment (MaxMind and DB-IP support).
// The mutator enriches the record; you then filter on the enriched geo.* fields.
// Mutator names cannot contain dots -- `ip | geoip.country_name` does NOT parse.
"ip | geoip_lookup exists"
"ip | geo exists"                        // `geo` and `geoip` are aliases
"geo.country_name = 'United States'"       // filter the enriched field
"geo.city_name = 'New York'"
"geo.continent_code = 'NA'"

// Configure with environment variables
// TQL_GEOIP_DB_PATH=/path/to/GeoLite2-City.mmdb
// TQL_GEOIP_MMDB_PATH=/usr/share/GeoIP/

Supported fields:

geo.country_name, geo.country_iso_code
geo.city_name
geo.location (lat/lon)
geo.continent_code, geo.continent_name
geo.region_name, geo.region_iso_code
geo.postal_code, geo.timezone

Performance: Uses memory-mapped I/O for efficient database access (200,000+ lookups/sec).

List Mutators

// Boolean aggregations
"tags | any = true"              // Check if any element is truthy
"flags | all = true"             // Check if all elements are truthy

// Numeric aggregations
"scores | avg > 80"              // Calculate average
"values | sum > 1000"            // Calculate sum
"prices | min >= 10"             // Find minimum value
"ratings | max <= 5"             // Find maximum value

// Example with nested arrays
"users.scores | avg > 75"         // Average of nested array
"metrics.values | sum between [100, 500]"  // Sum within range

Chaining Mutators

// Multiple transformations in sequence
"email | trim | lowercase contains '@example.com'"
"data | b64decode | lowercase = 'secret'"
"geo.country_name | lowercase = 'united states'"   // after `ip | geo()` enrichment

A mutator name TQL does not know is rejected at parse time. count, unique, first and last used to parse and then quietly return nothing; they are now parse errors in every runtime.

🏷️ Type Hints

A ::hint suffix tells TQL how to read a field's value before the comparison runs. It is applied after any field mutators and before the operator:

"value::number > 75"           // read as a number, then compare
"flag::string eq 'true'"       // read as a string, then compare
"f | trim::number > 75"        // after the mutator chain

These hints were inert until this release. type_hint was parsed, stored on the AST node, and read by no arm of the evaluator, so f::int eq 'Hello' answered identically to the un-hinted query. Every name now reaches a decision, and Rust gained the four names Python already had — date, geo, object and ip — so both runtimes accept the same sixteen:

Hint	Behaviour
`string`, `str`	render the value as a string
`integer`, `int`, `number`, `decimal`, `float`, `double`	numeric conversion
`boolean`, `bool`	boolean conversion
`ip`	validate as an IP address (and as a CIDR under the `cidr` operator)
`array`, `list`, `date`, `geo`, `object`	assert the field's shape; no value conversion

Three rules govern what happens when a hint meets a value it cannot read:

The record is skipped; the query is not aborted. value::number > 75 over [{v: 80}, {v: "abc"}, {v: 90}] matches 80 and 90.
A skipped record matches nothing, positively or negatively. f::number != 75 over "abc" is false, not true — likewise not_contains, not_startswith, not_endswith, not_in, not_cidr, and all four existence operators. f::int exists, f::int not exists, f::int is null and f::int is not null are all false over {"f": "Hello"}: f does not exist as an int, which is what the hint asked. An absent or null field is not a read failure, so f::int not exists over {} is unchanged.
A query-level not (...) does still invert a skip, the same way it already does for an absent field and the same way must_not answers on the cluster path.

An unrecognised hint name is a hard error, not a skip.

::string renders a boolean as true / false, not True / False — matching JSON, TQL's own boolean literals, and what OpenSearch stores. Scope is narrow on purpose: only a top-level boolean, and only under string/str. A boolean nested in a list or object still renders True (['a', True]), and error text still renders the offending value with the capitalised spelling.

A hint and a mutator chain may be written in EITHER order. f | trim::number > 75 and f::number | trim > 75 both parse, in both runtimes and in the TypeScript editor, and select the same records. This was a parity divergence — each grammar rejected the other's spelling — and it was closed in this release. If you find a page still describing one order as runtime-specific, that page is stale.

Type hints are honoured by the in-memory evaluator. The OpenSearch execution path ignores them: the cluster reads the value as the mapping says it is stored.

📊 Statistical Aggregations

TQL includes powerful stats functions for data analysis:

Available Functions

use tellaro_query_language::{StatsEvaluator, StatsQuery, AggregationSpec};
use std::collections::HashMap;

let evaluator = StatsEvaluator::new();
let records = vec![
    json!({"city": "NYC", "sales": 100, "product": "laptop"}),
    json!({"city": "LA", "sales": 150, "product": "phone"}),
    json!({"city": "NYC", "sales": 200, "product": "tablet"}),
];

// Count records
let query = StatsQuery {
    aggregations: vec![AggregationSpec {
        function: "count".to_string(),
        field: "*".to_string(),
        alias: Some("total".to_string()),
        params: HashMap::new(),
    }],
    group_by: vec![],
};
let result = evaluator.evaluate_stats(&records, &query)?;
// result["value"] = 3

// Sum with grouping
let query = StatsQuery {
    aggregations: vec![AggregationSpec {
        function: "sum".to_string(),
        field: "sales".to_string(),
        alias: Some("total_sales".to_string()),
        params: HashMap::new(),
    }],
    group_by: vec!["city".to_string()],
};
let result = evaluator.evaluate_stats(&records, &query)?;
// Groups by city: {"NYC": {"total_sales": 300}, "LA": {"total_sales": 150}}

CLI Stats Queries

# Simple aggregations
tql '| stats count()' data.jsonl
tql '| stats sum(revenue), avg(price)' sales.json

# Grouped analysis
tql '| stats count() by status' events.jsonl
tql '| stats sum(sales) by region, category' data.json

# Top N analysis
tql '| stats sum(revenue) top 10 by product' sales.json

# Combined filtering and stats
tql 'region = "west" | stats avg(revenue) by category' data.json

Aggregation Functions

count: Count records (count(*) or count(field))
sum: Sum numeric values
avg/average/mean: Calculate mean
min/max: Find minimum/maximum values
median/med: Calculate median
std / standard_deviation: Calculate standard deviation
percentile / p / pct: Calculate percentiles
distinct / unique / values: Return unique values

The full list is the agg_func_name rule in tql/src/parser/grammar.pest, but six names in that rule are unreachable and will not parse: stddev, percentiles, percentile_rank, percentile_ranks, pct_rank and pct_ranks. agg_func_name is a PEG ordered choice that lists the shorter alternative first, so std consumes the front of stddev, percentile consumes percentiles and percentile_rank(s), and pct consumes pct_rank(s). Use the names listed above. (There is also no unique_count in Rust, though the Python implementation does accept it.)

🎯 API Reference

Basic Usage

use tellaro_query_language::Tql;
use serde_json::json;

// Create TQL instance
let tql = Tql::new();

// Or with custom depth limits
let tql = Tql::with_max_depth(200);

// Query records
let records = vec![
    json!({"name": "Alice", "age": 30, "city": "NYC"}),
    json!({"name": "Bob", "age": 25, "city": "LA"}),
];

// Execute query
let results = tql.query(&records, "age > 27").unwrap();
println!("Found {} matching records", results.len());

// Count matches
let count = tql.count(&records, "city = 'NYC'").unwrap();
println!("NYC residents: {}", count);

// Check single record
let user = json!({"age": 30, "status": "active"});
if tql.matches(&user, "age >= 18 AND status = 'active'").unwrap() {
    println!("Valid adult user");
}

Query Pre-compilation

For queries executed multiple times, parse once and reuse the AST:

use tellaro_query_language::{Tql, TqlEvaluator};

let tql = Tql::new();

// Parse query once
let ast = tql.parse("age > 25 AND status = 'active'").unwrap();

// Reuse AST for multiple datasets
let evaluator = TqlEvaluator::new();
let results1 = evaluator.filter(&ast, &dataset1).unwrap();
let results2 = evaluator.filter(&ast, &dataset2).unwrap();

Error Handling

use tellaro_query_language::{Tql, TqlError};

let tql = Tql::new();

match tql.query(&records, "invalid query syntax") {
    Ok(results) => println!("Found {} records", results.len()),
    // SyntaxError and ParseError are STRUCT variants, not tuple variants
    Err(TqlError::SyntaxError { message, .. }) => eprintln!("Syntax error: {}", message),
    Err(TqlError::ParseError { message, position, .. }) => {
        eprintln!("Parse error at {}: {}", position, message)
    }
    Err(TqlError::ExecutionError(msg)) => eprintln!("Execution error: {}", msg),
    Err(e) => eprintln!("Error: {}", e),
}

⚡ Performance

Benchmarks

Unsubstantiated. The figures below are carried over from an earlier revision and could not be reproduced: this crate ships no benchmark harness (no benches/ directory and no [[bench]] target), so cargo bench runs nothing. Treat them as rough historical claims, not measurements. Removing them, or landing a Criterion benchmark that produces them, is tracked work.

Rust Implementation:

In-memory queries: ~3,000,000 records/sec
File parsing (JSON): ~150MB/sec
GeoIP lookups: ~200,000 lookups/sec (memory-mapped)
DNS lookups: ~10,000 lookups/sec (with caching)
Large file streaming: Process 50MB in ~200ms

vs Python Implementation:

300x faster for file processing
500x faster for GeoIP lookups (memory-mapped vs Python)
100x faster for DNS lookups (async + caching)

Performance Features

Zero-copy deserialization where possible
Memory-mapped I/O for GeoIP databases
In-memory caching for DNS and GeoIP results
Streaming file processing with no memory overhead
Parallel query evaluation (planned)

Optimization Tips

// Pre-compile queries for reuse
let ast = tql.parse("age > 25").unwrap();
let results1 = evaluator.filter(&ast, &dataset1).unwrap();
let results2 = evaluator.filter(&ast, &dataset2).unwrap();

// Use streaming for large files
let file = File::open("large.jsonl")?;
let reader = BufReader::new(file);
for line in reader.lines() {
    let record: Value = serde_json::from_str(&line?)?;
    if tql.matches(&record, "level = 'ERROR'")? {
        // Process match without loading entire file
    }
}

🗺️ Roadmap

✅ Implemented Features

✅ Core query engine with all operators
✅ 25+ field mutators (string, encoding, network, DNS, GeoIP, list)
✅ Statistical aggregations with grouping
✅ File support (JSON, JSONL, CSV) with CLI
✅ OpenSearch backend with automatic DSL translation
✅ Memory-mapped GeoIP lookups
✅ DNS resolution with caching
✅ High-performance streaming

🚧 In Progress

🚧 OpenSearch stats aggregation translation
🚧 Post-processing for complex mutator chains
🚧 Additional hash functions (SHA1, SHA512)

📋 Planned Features

📋 Parallel record evaluation
📋 Query optimization engine
📋 JSON parsing mutator
📋 Timestamp conversion mutators
📋 PostgreSQL/MySQL backends
📋 Custom mutator plugins via traits

🔮 Future Considerations

🔮 Distributed query execution
🔮 Query result caching
🔮 Real-time data streaming
🔮 WASM compilation for browser usage

🔧 Development

Setup

# Clone repository
git clone https://github.com/tellaro/tellaro-query-language.git
cd tellaro-query-language/tql

# Build
cargo build

# Run tests
cargo test

# Build release
cargo build --release

# Build with OpenSearch feature
cargo build --features opensearch

Testing

# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run integration tests (requires OpenSearch)
export OPENSEARCH_HOSTS=http://localhost:9200
export OPENSEARCH_USERNAME=admin
export OPENSEARCH_PASSWORD=admin
export OPENSEARCH_INTEGRATION_TEST=true
cargo test --features opensearch -- --ignored --test-threads=1

Code Quality

# Format code
cargo fmt

# Linting
cargo clippy -- -D warnings

# Check compilation
cargo check

🤝 Contributing

Contributions are welcome! See docs/developer/contributing.md in the repository root for guidelines.

📄 License

Tellaro Query Language (TQL) is source-available software with specific usage terms:

✅ Permitted Uses:

Personal use (individual, non-commercial)
Organizational use (within your company/organization)
Integration into your applications and services
Internal tools and automation

❌ Restricted Uses:

Creating derivative query language products
Commercial redistribution or resale
Offering TQL-based commercial services to third parties
Using source code to build competing products

For commercial licensing inquiries, contact: support@tellaro.io

See LICENSE for complete terms and conditions.

🔗 Related Projects

TQL Python - Python implementation with rich ecosystem
Tellaro Platform - Security operations platform using TQL

💬 Support

Issues: GitHub Issues
Documentation: docs.rs/tellaro-query-language
Email: support@tellaro.io

🌟 Advanced Examples

Security Log Analysis

use tellaro_query_language::Tql;
use serde_json::json;

let tql = Tql::new();
let logs = vec![
    json!({
        "timestamp": "2024-01-15T10:30:00Z",
        "source_ip": "192.168.1.100",
        "url": "hxxp://malicious[.]com/payload",
        "severity": "high",
        "tags": ["suspicious", "malware"]
    }),
    json!({
        "timestamp": "2024-01-15T10:31:00Z",
        "source_ip": "10.0.0.50",
        "url": "https://safe-site.com",
        "severity": "low",
        "tags": ["normal"]
    }),
];

// Find high-severity events with malicious indicators
let query = r#"
    severity in ['high', 'critical'] AND
    source_ip | is_private = true AND
    (ANY tags = 'malware' OR url | refang contains 'malicious')
"#;

let results = tql.query(&logs, query).unwrap();
assert_eq!(results.len(), 1);

E-commerce Product Search

let products = vec![
    json!({
        "name": "Laptop Pro 15",
        "price": 1299.99,
        "tags": ["electronics", "computers", "premium"],
        "rating": {"average": 4.5, "count": 128},
        "in_stock": true
    }),
    json!({
        "name": "Budget Mouse",
        "price": 9.99,
        "tags": ["electronics", "accessories"],
        "rating": {"average": 3.8, "count": 45},
        "in_stock": false
    }),
];

// Find in-stock electronics with good ratings under $1500
let query = r#"
    in_stock = true AND
    price < 1500 AND
    rating.average >= 4.0 AND
    ANY tags = 'electronics'
"#;

let results = tql.query(&products, query).unwrap();
assert_eq!(results.len(), 1);

GeoIP Enrichment Pipeline

// Set GeoIP database path
std::env::set_var("TQL_GEOIP_DB_PATH", "/usr/share/GeoIP/GeoLite2-City.mmdb");

let tql = Tql::new();
let events = vec![
    json!({"ip": "8.8.8.8", "event": "login"}),
    json!({"ip": "1.1.1.1", "event": "api_call"}),
];

// Query with GeoIP enrichment
// Enrich, then filter on the enriched field
let query = "ip | geo exists AND geo.country_name = 'United States'";
let results = tql.query(&events, query).unwrap();

// Results include enriched geo data
println!("{}", serde_json::to_string_pretty(&results[0]).unwrap());
// {
//   "ip": "8.8.8.8",
//   "event": "login",
//   "geo": {
//     "country_name": "United States",
//     "country_iso_code": "US",
//     "city_name": "Mountain View",
//     "location": {"lat": 37.386, "lon": -122.0838}
//   }
// }

Made with ❤️ by the Tellaro Team

tellaro-query-language 2.0.0