Expand description
The evalhub query language: a small, typed filter DSL whose vocabulary is projected from the record schema.
POST /cards/query and POST /evals/query accept a JSON query
(envelope in evalhub_schema::query). This crate turns that JSON into
something a backend can execute, in three stages, none of which knows
SQL:
JSON ──▶ grammar::parse structure: and / or / not / leaf { path, op, value | match }
──▶ typecheck::check every path exists, has a type, and the op fits it
──▶ ir::Query backend-independent tree the store compiles to SQLA query that exercises every construct:
{
"where": {"and": [
{"path": "model.id", "op": "eq", "value": "qwen3.6-32b"},
{"path": "generation.temperature", "op": "lte", "value": 0.2},
{"path": "results", "op": "any", "match": {"metric": "core/pass_rate", "value": {"op": "gte", "value": 0.5}}},
{"path": "relations", "op": "any", "match": {"type": "core/uses_eval", "to": "alice/single2-k4"}},
{"path": "fingerprint.generation", "op": "eq", "value": "…"},
{"path": "ext.alice/qwen-loop.rung", "op": "eq", "value": 4}
]},
"sort": [{"path": "results[core/pass_rate].value", "dir": "desc"}],
"limit": 50, "cursor": null,
"expand": ["relations", "badges", "fingerprints", "changed"],
"version": "latest"
}§The grammar is the specification
The filter AST is a dsl-kit enum, and the conformance check, the typed
builder and the published JSON Schema of the where value all come from
it. The operator vocabulary in particular is declared once, as
grammar::ScalarOp: adding an operator is a variant plus its row in
the type table, and no parser arm changes.
What is hand-written is the lowering from the wire shape
({"and": [...]}) into the tagged shape dsl-kit’s JSON front end reads
({"type": "And", …}), which names the five node shapes. See the
grammar module doc for why that is a shape concern rather than a
vocabulary one.
§The path table is projected from the schema
Which paths are legal (model.id, generation.temperature,
results[core/pass_rate].value, fingerprint.model,
ext.alice/qwen-loop.rung) and what type each has is not written by
hand either. At start-up typecheck walks the committed JSON Schema
from evalhub_schema and builds the table: every leaf of a facet becomes
a path with its JSON type, fingerprint.{facet} is added for each facet,
results / relations / attachments become array paths that accept
any, and registered ext_schemas contribute typed ext. paths. A key
added to the schema is queryable in the same commit.
§Type rules
| Operator | Accepts |
|---|---|
eq, ne, in | any scalar |
gt, gte, lt, lte | number, string (lexical) |
prefix, contains | string |
exists | any path |
any with match | array of objects |
A mismatch is 422 type_mismatch. A path not in the table is
422 unknown_path — never an empty result, because an empty result for
a typo looks like data.
§Index awareness
The type check also knows which paths are indexed: the sixteen facet
keys the migration materialises as generated columns, the fingerprints,
and the results / relations / attachments side tables, plus ext
paths whose ext_schema is applied. Every other path — an
unregistered ext key, and equally a facet key with no column of its
own — accepts eq and exists only, which is what the GIN index over
the JSON body answers; any other operator is 422 not_indexed with a
hint naming the registry. This keeps a query from silently becoming a
sequential scan.
§Sorting and cursors
sort accepts the same paths. results[{metric}].value sorts in the
direction the metric’s registry entry declares (lower_is_better); an
unregistered metric cannot be sorted on, because the hub does not know
which way is up. Pagination is keyset (sort key + version_id), and the
IR carries the decoded cursor so the store can emit the WHERE (k, id) > (?, ?) clause; the cursor’s opaque, signed encoding is the server’s
business.
§The second target: the run projection
GET /evals/{ns}/{name}/runs reads one row per run of one Eval, joined
with the run_results of the Cards the request names (cards=). It
speaks the same grammar — where, sort, limit, cursor — against
a different path table, and compiles to a different IR:
JSON ──▶ grammar::parse (unchanged)
──▶ typecheck::check against PathTable::for_runs(cards)
──▶ ir::RunQuery { cards, filter, sort, limit, cursor: RunCursor }A second target because a run is not a record. An Eval is a versioned
header plus runs that belong to the record, not to a version, so the
header’s schema (what the record table is projected from) has no
runs key, and a record query cannot reach a run. A Card judges runs
in run_results, within its used set (the runs of the Eval it
used); the projection is where the two meet, one row per run with the
judgements of each named Card. What the page reports beside the rows
(per Card the used-set hash, the one recorded when the Card was
posted, and the runs changed since; each row’s content_hash) is
computed by the store with evalhub_core::run and is not a path of
this table, and neither is the record’s runs_hash, which the Eval’s
own GET and every run write report. Nor is anything
about who is asking: runs follow their Eval’s visibility, judgements
their Card’s, and the limits (page size 1–200, the batch size, the
number of run_results per Card) are the server’s; a query that
compiles here says nothing about what the caller may see. None of it
writes: header versions are write-once (the one exception, the 0.2.0
data migration, is evalhub_store’s) and runs are written through
the run endpoints, not through this crate.
PathTable::for_runs is built by hand, not from a record schema: the
run’s columns (run_id, status, error.kind, started_at,
ended_at), the keys of its six facets (the Eval header’s key set,
model.id and so on), fingerprint.{facet}, metrics[{ns}/{name}],
results[{card}][{metric}].value and .label for the named Cards, and
ext.* through the same PathTable::with_ext. Parameters are
bracketed, as in results[core/pass_rate].value, because a slug may
contain . and /; the dotted forms are refused with a hint. A Card
the request did not name is unknown_path, and so is a malformed metric
id; a metric the registry does not know is accepted, because a run’s
metrics are.
{
"where": {"and": [
{"path": "status", "op": "eq", "value": "ok"},
{"path": "metrics[core/tokens_out]", "op": "lt", "value": 4000},
{"path": "results[alice/judge][core/pass].value", "op": "gte", "value": 1}
]},
"sort": [{"path": "metrics[core/duration_ms]", "dir": "asc"}]
}Every path of the projection sorts and ranges as its type allows
(a boolean facet key does not range), facet keys included, except an
unregistered ext key, which answers eq and exists only because
nothing declares its type. The projection is scoped to one Eval and
cannot grow into a scan of the hub; the not_indexed rule above is
about record queries.
Paging is keyset on (sort key, run_id) with ir::RunCursor, which
the server signs exactly as it signs ir::Cursor. Which Eval, whether
archived and deleted runs are included (include=archived,deleted) and
which Cards a caller may name are the server’s decisions, not the
DSL’s. The typecheck module doc has the full table.
§Modules
Re-exports§
pub use grammar::Filter;pub use grammar::MatchTerm;pub use grammar::ScalarOp;pub use grammar::parse;pub use grammar::request_schema;pub use ir::CardRef;pub use ir::Query;pub use ir::RunCursor;pub use ir::RunQuery;pub use typecheck::ExtSchema;pub use typecheck::Indexed;pub use typecheck::PathInfo;pub use typecheck::PathTable;pub use typecheck::check;pub use typecheck::compile;pub use typecheck::compile_runs;