Skip to main content

Module typecheck

Module typecheck 

Source
Expand description

The schema-derived path table and the type check.

PathTable is built once per process from the committed JSON Schema in evalhub_schema plus the currently applied ext_schemas from the registry. It maps each legal query path to where its value lives, what type it has, and which operators the storage can actually serve:

Path formTypeServed byOperators
{facet}.{key} with a columnfrom the schema leafgenerated columnall that suit the type
{facet}.{key} without onefrom the schema leafGIN index on bodyeq, exists
fingerprint.{facet}stringfingerprints tableeq, ne, in, exists
results, relations, attachmentsarray of objectown tablesany
results[{metric}].value (sort)numberresults tablesort only
ext.{ns}.{key} registeredfrom the ext_schemaexpression indexall that suit the type
ext.{ns}.{key} unregisteredunknownGIN index on bodyeq, exists
title, producer.name, …stringgenerated columnall that suit the type

The row that is easy to get wrong is the second one. Only sixteen facet keys have a generated column (the migration lists them, and GENERATED_COLUMNS mirrors that list with a test that fails if the two disagree). Every other key of the record is readable and equality- searchable through the GIN index over body, but ordering or ranging on it would be a sequential scan, so the check refuses with not_indexed and names the registry. That is the same answer an unregistered ext path gets, and for the same reason.

runs is deliberately absent: an Eval’s runs are rows of their own, not a key of the header’s body, so a record query cannot reach them. A question about runs is asked of the run projection, which has its own table (below).

The check walks the parsed tree and, for each leaf, looks the path up, confirms the operator is legal for the type, confirms the literal is of that type, and confirms the operator is served by the path’s index. Failures are collected into 422 { errors[] } with codes unknown_path, type_mismatch, not_indexed, each carrying a JSON pointer into the submitted request.

The table is rebuilt when an ext_schema transitions to applied; the server holds it behind an ArcSwap-style handle so a query in flight sees a consistent table.

§The run projection’s table

GET /evals/{ns}/{name}/runs reads one row per run of one Eval, joined with the run_results of the Cards the request names. Its table is PathTable::for_runs, built by hand rather than from a record schema, because a row of the projection is not a record: it is a run’s columns, its side tables, and a join.

Path formTypeResolves to
run_id, status, error.kindstringa column of runs
started_at, ended_atstring (RFC 3339)a column of runs
{facet}.{key} (model.id, …)from the schema leafthe run’s body
fingerprint.{facet}stringrun_fingerprints
metrics[{ns}/{name}]numberrun_metrics
results[{card}][{metric}].valuenumberthe Card’s run_results
results[{card}][{metric}].labelstringthe Card’s run_results
ext.{ns}.{key}as for recordsthe run’s body
  • The facet keys are the Eval header’s: the leaves of the six Eval facets in RecordKind::Eval.schema() (a run carries the same facet types, evalhub_schema::run). There is no grading.
  • metrics[…] takes any well-formed metric id ({ns}/{name}, the registry id rule); the registry is not consulted, because a run’s metrics are accepted whether or not the registry knows them, and so they must be searchable the same way. A malformed id is unknown_path.
  • results[{card}]… exists only for the Cards named in the request (CardRef, {ns}/{name}). A Card the request did not name is unknown_path, with a hint to name it: joining a Card the caller did not ask for would change the projection behind the caller’s back. Which Cards a caller may name at all is the server’s decision, made before the table is built.
  • Parameters use the bracket form the record table’s results[{metric}].value sort key uses. The dotted forms (metrics.core/tokens_out, results.alice/g.core/pass.value) are refused with a hint naming the bracket form: a slug may contain ., so a dotted path cannot say where a parameter ends. The bracket form needs no escaping because ids and Card references are made of [a-z0-9._-] and one /, never [ or ].
  • ext.* comes through the same PathTable::with_ext as for records; in the projection its paths are below the run’s own ext.
  • There are no array paths and so no any: a run’s attachments[] and artifacts[] are not searchable here, and a Card’s judgements are addressed by Card and metric rather than ranged over.

Index awareness is different here. Every path of the projection is served in full (every operator its type allows, and sortable), facet keys included, although the run’s facets have no generated column. The record table refuses ranges on such keys because a record query spans every record of the hub, and a range the index cannot answer grows with the hub. The projection is always scoped to one Eval (runs is keyed by (record, run_id)), so the worst case is a scan of one Eval’s runs, which the page limit and the runs:batch limits bound. An ext key with no registered schema still answers only eq and exists, not for the index but for the type: nothing declares what it holds, so a range on it has no meaning.

Structs§

ExtSchema
A registered extension schema, as the registry holds it.
PathTable
The set of legal query paths with their types and index status, projected from the record schema and the applied ext_schemas, or built for the run projection by PathTable::for_runs.

Enums§

Indexed
How much of the operator set a path’s storage can serve.
PathInfo
What a legal query path denotes.

Constants§

GENERATED_COLUMNS
The generated columns of versions, as JSON path and column name.

Functions§

check
Type-check a parsed filter against a path table.
compile
Parse, check and assemble a whole request into the IR the store runs.
compile_runs
Parse, check and assemble a request for the run projection (GET /evals/{ns}/{name}/runs) into the IR the store runs.