Expand description
The schema-derived path table and the type check.
PathTable is built once per process from the committed JSON Schema in
evalhub_schema plus the currently applied ext_schemas from the
registry. It maps each legal query path to where its value lives, what
type it has, and which operators the storage can actually serve:
| Path form | Type | Served by | Operators |
|---|---|---|---|
{facet}.{key} with a column | from the schema leaf | generated column | all that suit the type |
{facet}.{key} without one | from the schema leaf | GIN index on body | eq, exists |
fingerprint.{facet} | string | fingerprints table | eq, ne, in, exists |
results, relations, attachments | array of object | own tables | any |
results[{metric}].value (sort) | number | results table | sort only |
ext.{ns}.{key} registered | from the ext_schema | expression index | all that suit the type |
ext.{ns}.{key} unregistered | unknown | GIN index on body | eq, exists |
title, producer.name, … | string | generated column | all that suit the type |
The row that is easy to get wrong is the second one. Only sixteen facet
keys have a generated column (the migration lists them, and
GENERATED_COLUMNS mirrors that list with a test that fails if the two
disagree). Every other key of the record is readable and equality-
searchable through the GIN index over body, but ordering or ranging on
it would be a sequential scan, so the check refuses with not_indexed
and names the registry. That is the same answer an unregistered ext
path gets, and for the same reason.
runs is deliberately absent: an Eval’s runs are rows of their own, not
a key of the header’s body, so a record query cannot reach them. A
question about runs is asked of the run projection, which has its own
table (below).
The check walks the parsed tree and, for each leaf, looks the path up,
confirms the operator is legal for the type, confirms the literal is of
that type, and confirms the operator is served by the path’s index.
Failures are collected into 422 { errors[] } with codes
unknown_path, type_mismatch, not_indexed, each carrying a JSON
pointer into the submitted request.
The table is rebuilt when an ext_schema transitions to applied; the
server holds it behind an ArcSwap-style handle so a query in flight
sees a consistent table.
§The run projection’s table
GET /evals/{ns}/{name}/runs reads one row per run of one Eval, joined
with the run_results of the Cards the request names. Its table is
PathTable::for_runs, built by hand rather than from a record
schema, because a row of the projection is not a record: it is a run’s
columns, its side tables, and a join.
| Path form | Type | Resolves to |
|---|---|---|
run_id, status, error.kind | string | a column of runs |
started_at, ended_at | string (RFC 3339) | a column of runs |
{facet}.{key} (model.id, …) | from the schema leaf | the run’s body |
fingerprint.{facet} | string | run_fingerprints |
metrics[{ns}/{name}] | number | run_metrics |
results[{card}][{metric}].value | number | the Card’s run_results |
results[{card}][{metric}].label | string | the Card’s run_results |
ext.{ns}.{key} | as for records | the run’s body |
- The facet keys are the Eval header’s: the leaves of the six Eval
facets in
RecordKind::Eval.schema()(a run carries the same facet types,evalhub_schema::run). There is nograding. metrics[…]takes any well-formed metric id ({ns}/{name}, the registry id rule); the registry is not consulted, because a run’s metrics are accepted whether or not the registry knows them, and so they must be searchable the same way. A malformed id isunknown_path.results[{card}]…exists only for the Cards named in the request (CardRef,{ns}/{name}). A Card the request did not name isunknown_path, with a hint to name it: joining a Card the caller did not ask for would change the projection behind the caller’s back. Which Cards a caller may name at all is the server’s decision, made before the table is built.- Parameters use the bracket form the record table’s
results[{metric}].valuesort key uses. The dotted forms (metrics.core/tokens_out,results.alice/g.core/pass.value) are refused with a hint naming the bracket form: a slug may contain., so a dotted path cannot say where a parameter ends. The bracket form needs no escaping because ids and Card references are made of[a-z0-9._-]and one/, never[or]. ext.*comes through the samePathTable::with_extas for records; in the projection its paths are below the run’s ownext.- There are no array paths and so no
any: a run’sattachments[]andartifacts[]are not searchable here, and a Card’s judgements are addressed by Card and metric rather than ranged over.
Index awareness is different here. Every path of the projection is
served in full (every operator its type allows, and sortable), facet
keys included, although the run’s facets have no generated column. The
record table refuses ranges on such keys because a record query spans
every record of the hub, and a range the index cannot answer grows with
the hub. The projection is always scoped to one Eval (runs is keyed
by (record, run_id)), so the worst case is a scan of one Eval’s runs,
which the page limit and the runs:batch limits bound. An ext key
with no registered schema still answers only eq and exists, not for
the index but for the type: nothing declares what it holds, so a range
on it has no meaning.
Structs§
- ExtSchema
- A registered extension schema, as the registry holds it.
- Path
Table - The set of legal query paths with their types and index status,
projected from the record schema and the applied
ext_schemas, or built for the run projection byPathTable::for_runs.
Enums§
- Indexed
- How much of the operator set a path’s storage can serve.
- Path
Info - What a legal query path denotes.
Constants§
- GENERATED_
COLUMNS - The generated columns of
versions, as JSON path and column name.
Functions§
- check
- Type-check a parsed filter against a path table.
- compile
- Parse, check and assemble a whole request into the IR the store runs.
- compile_
runs - Parse, check and assemble a request for the run projection
(
GET /evals/{ns}/{name}/runs) into the IR the store runs.