evalhub_query/lib.rs
1//! The evalhub query language: a small, typed filter DSL whose vocabulary is
2//! projected from the record schema.
3//!
4//! `POST /cards/query` and `POST /evals/query` accept a JSON query
5//! (envelope in `evalhub_schema::query`). This crate turns that JSON into
6//! something a backend can execute, in three stages, none of which knows
7//! SQL:
8//!
9//! ```text
10//! JSON ──▶ grammar::parse structure: and / or / not / leaf { path, op, value | match }
11//! ──▶ typecheck::check every path exists, has a type, and the op fits it
12//! ──▶ ir::Query backend-independent tree the store compiles to SQL
13//! ```
14//!
15//! A query that exercises every construct:
16//!
17//! ```json
18//! {
19//! "where": {"and": [
20//! {"path": "model.id", "op": "eq", "value": "qwen3.6-32b"},
21//! {"path": "generation.temperature", "op": "lte", "value": 0.2},
22//! {"path": "results", "op": "any", "match": {"metric": "core/pass_rate", "value": {"op": "gte", "value": 0.5}}},
23//! {"path": "relations", "op": "any", "match": {"type": "core/uses_eval", "to": "alice/single2-k4"}},
24//! {"path": "fingerprint.generation", "op": "eq", "value": "…"},
25//! {"path": "ext.alice/qwen-loop.rung", "op": "eq", "value": 4}
26//! ]},
27//! "sort": [{"path": "results[core/pass_rate].value", "dir": "desc"}],
28//! "limit": 50, "cursor": null,
29//! "expand": ["relations", "badges", "fingerprints", "changed"],
30//! "version": "latest"
31//! }
32//! ```
33//!
34//! # The grammar is the specification
35//!
36//! The filter AST is a `dsl-kit` enum, and the conformance check, the typed
37//! builder and the published JSON Schema of the `where` value all come from
38//! it. The operator vocabulary in particular is declared once, as
39//! [`grammar::ScalarOp`]: adding an operator is a variant plus its row in
40//! the type table, and no parser arm changes.
41//!
42//! What *is* hand-written is the lowering from the wire shape
43//! (`{"and": [...]}`) into the tagged shape `dsl-kit`'s JSON front end reads
44//! (`{"type": "And", …}`), which names the five node shapes. See the
45//! [`grammar`] module doc for why that is a shape concern rather than a
46//! vocabulary one.
47//!
48//! # The path table is projected from the schema
49//!
50//! Which paths are legal (`model.id`, `generation.temperature`,
51//! `results[core/pass_rate].value`, `fingerprint.model`,
52//! `ext.alice/qwen-loop.rung`) and what type each has is not written by
53//! hand either. At start-up [`typecheck`] walks the committed JSON Schema
54//! from `evalhub_schema` and builds the table: every leaf of a facet becomes
55//! a path with its JSON type, `fingerprint.{facet}` is added for each facet,
56//! `results` / `relations` / `attachments` become array paths that accept
57//! `any`, and registered `ext_schemas` contribute typed `ext.` paths. A key
58//! added to the schema is queryable in the same commit.
59//!
60//! # Type rules
61//!
62//! | Operator | Accepts |
63//! | ----------------------------------- | -------------------------------- |
64//! | `eq`, `ne`, `in` | any scalar |
65//! | `gt`, `gte`, `lt`, `lte` | number, string (lexical) |
66//! | `prefix`, `contains` | string |
67//! | `exists` | any path |
68//! | `any` with `match` | array of objects |
69//!
70//! A mismatch is `422 type_mismatch`. A path not in the table is
71//! `422 unknown_path` — never an empty result, because an empty result for
72//! a typo looks like data.
73//!
74//! # Index awareness
75//!
76//! The type check also knows which paths are indexed: the sixteen facet
77//! keys the migration materialises as generated columns, the fingerprints,
78//! and the `results` / `relations` / `attachments` side tables, plus `ext`
79//! paths whose `ext_schema` is `applied`. Every other path — an
80//! unregistered `ext` key, and equally a facet key with no column of its
81//! own — accepts `eq` and `exists` only, which is what the GIN index over
82//! the JSON body answers; any other operator is `422 not_indexed` with a
83//! hint naming the registry. This keeps a query from silently becoming a
84//! sequential scan.
85//!
86//! # Sorting and cursors
87//!
88//! `sort` accepts the same paths. `results[{metric}].value` sorts in the
89//! direction the metric's registry entry declares (`lower_is_better`); an
90//! unregistered metric cannot be sorted on, because the hub does not know
91//! which way is up. Pagination is keyset (`sort key + version_id`), and the
92//! IR carries the decoded cursor so the store can emit the `WHERE (k, id) >
93//! (?, ?)` clause; the cursor's opaque, signed encoding is the server's
94//! business.
95//!
96//! # The second target: the run projection
97//!
98//! `GET /evals/{ns}/{name}/runs` reads one row per run of one Eval, joined
99//! with the `run_results` of the Cards the request names (`cards=`). It
100//! speaks the same grammar — `where`, `sort`, `limit`, `cursor` — against
101//! a different path table, and compiles to a different IR:
102//!
103//! ```text
104//! JSON ──▶ grammar::parse (unchanged)
105//! ──▶ typecheck::check against PathTable::for_runs(cards)
106//! ──▶ ir::RunQuery { cards, filter, sort, limit, cursor: RunCursor }
107//! ```
108//!
109//! A second target because a run is not a record. An Eval is a versioned
110//! header plus runs that belong to the record, not to a version, so the
111//! header's schema (what the record table is projected from) has no
112//! `runs` key, and a record query cannot reach a run. A Card judges runs
113//! in `run_results`, within its *used set* (the runs of the Eval it
114//! used); the projection is where the two meet, one row per run with the
115//! judgements of each named Card. What the page reports beside the rows
116//! (per Card the used-set hash, the one recorded when the Card was
117//! posted, and the runs changed since; each row's `content_hash`) is
118//! computed by the store with `evalhub_core::run` and is not a path of
119//! this table, and neither is the record's `runs_hash`, which the Eval's
120//! own `GET` and every run write report. Nor is anything
121//! about who is asking: runs follow their Eval's visibility, judgements
122//! their Card's, and the limits (page size 1–200, the batch size, the
123//! number of `run_results` per Card) are the server's; a query that
124//! compiles here says nothing about what the caller may see. None of it
125//! writes: header versions are write-once (the one exception, the 0.2.0
126//! data migration, is `evalhub_store`'s) and runs are written through
127//! the run endpoints, not through this crate.
128//!
129//! [`PathTable::for_runs`] is built by hand, not from a record schema: the
130//! run's columns (`run_id`, `status`, `error.kind`, `started_at`,
131//! `ended_at`), the keys of its six facets (the Eval header's key set,
132//! `model.id` and so on), `fingerprint.{facet}`, `metrics[{ns}/{name}]`,
133//! `results[{card}][{metric}].value` and `.label` for the named Cards, and
134//! `ext.*` through the same [`PathTable::with_ext`]. Parameters are
135//! bracketed, as in `results[core/pass_rate].value`, because a slug may
136//! contain `.` and `/`; the dotted forms are refused with a hint. A Card
137//! the request did not name is `unknown_path`, and so is a malformed metric
138//! id; a metric the registry does not know is accepted, because a run's
139//! metrics are.
140//!
141//! ```json
142//! {
143//! "where": {"and": [
144//! {"path": "status", "op": "eq", "value": "ok"},
145//! {"path": "metrics[core/tokens_out]", "op": "lt", "value": 4000},
146//! {"path": "results[alice/judge][core/pass].value", "op": "gte", "value": 1}
147//! ]},
148//! "sort": [{"path": "metrics[core/duration_ms]", "dir": "asc"}]
149//! }
150//! ```
151//!
152//! Every path of the projection sorts and ranges as its type allows
153//! (a boolean facet key does not range), facet keys included, except an
154//! unregistered `ext` key, which answers `eq` and `exists` only because
155//! nothing declares its type. The projection is scoped to one Eval and
156//! cannot grow into a scan of the hub; the `not_indexed` rule above is
157//! about record queries.
158//! Paging is keyset on `(sort key, run_id)` with [`ir::RunCursor`], which
159//! the server signs exactly as it signs [`ir::Cursor`]. Which Eval, whether
160//! archived and deleted runs are included (`include=archived,deleted`) and
161//! which Cards a caller may name are the server's decisions, not the
162//! DSL's. The [`typecheck`] module doc has the full table.
163//!
164//! # Modules
165//!
166//! - [`grammar`] — the `dsl-kit` definition and the parser it yields.
167//! - [`typecheck`] — the path tables (schema-derived for records, built by
168//! hand for runs) and the type rules.
169//! - [`ir`] — the trees handed to the store, [`Query`] and [`RunQuery`].
170
171pub mod grammar;
172pub mod ir;
173pub mod typecheck;
174
175pub use grammar::{Filter, MatchTerm, ScalarOp, parse, request_schema};
176pub use ir::{CardRef, Query, RunCursor, RunQuery};
177pub use typecheck::{ExtSchema, Indexed, PathInfo, PathTable, check, compile, compile_runs};