Expand description
Record types for evalhub, and the source of its public contract.
evalhub is a hosting service for LLM evaluation results. This crate holds the shape of what the hub accepts and returns: the two record kinds, the facets they share, the query language’s request and response, and the error envelope. Everything else in the workspace is built on these types.
§What lives here and what does not
This crate is types only. It has no I/O, no validation beyond what
serde enforces structurally, no knowledge of Postgres or HTTP. The
semantic rules (counts must add up, attachment references must resolve,
integers must fit in 2^53) live in evalhub_core; the transport lives in
evalhub_server. Keeping the types free of behaviour is what lets them be
the contract: a client that deserialises schemas/card.json sees exactly
what the server sees.
§The contract is generated from these types, not the other way round
The Rust types are the source. Each public record type derives
schemars::JsonSchema, and the JSON Schema (draft 2020-12) is generated
from it and committed under schemas/. A snapshot test asserts that the
generated schema equals the committed file, so a change to a type is not
done until the schema in the same commit has changed with it. The OpenAPI
3.1 document served by evalhub_server embeds the same schemas, produced by
the same derive, so there is one description of a Card and it is this
crate.
The alternative — hand-written JSON Schema with Rust types generated from
it — was rejected: generated Rust cannot carry the serde attributes and
doc comments the server needs, and two hand-maintained documents (record
schema and OpenAPI) drift.
Doc comments on these types are public. They become the description in
the served schema and in openapi.json, so they are written for the person
writing a client, not for the person reading this crate.
§The two record kinds
A card::Card says what was measured, how, and what the score was. An
eval::Eval is the material: a set of runs, prompts, tasks or traces
that a Card was measured from. Both are published under a name,
{ns}/{name}, as an append-only sequence of immutable versions (the one
exception, the 0.2.0 data migration that rewrote stored
evalhub.eval/1.0 bodies into 2.0 headers, is evalhub_store’s). The
relation between them (core/uses_eval) is what makes comparison possible:
two Cards that point at the same Eval version measured the same thing.
An Eval’s versioned body is a header. Its runs (run::Run) are rows
of the Eval record, outside every version: writing a run does not append
a header version. A run records what happened and what was measured
directly; the verdict on it is the Card’s (card::Card::run_results).
The runs a Card judged are its used set, per Eval it uses: the
attrs.runs of its core/uses_eval relations, or, without them, every
run neither archived nor deleted when the Card is posted (common,
card). Four hashes tie the pieces together, all computed by the
hub and none sent by the client: the header’s content_hash, each
run’s content_hash, the record’s runs_hash over every run, and a
used-set hash over the runs a Card used; the formulas are
evalhub_core’s. Runs have no visibility of their own and follow their
Eval; a Card’s run_results follow the Card, except that a reader who
may not see an Eval is never shown the entries that name it. How many
runs a batch and how many run_results a Card may carry are the
server’s limits, not constraints of these types.
A Card has no kind field. Whether a score came from a judge, an
aggregation, or was transcribed from a paper is not a kind of Card; those
are properties of how it was graded (facet::Grading), how it was
aggregated (results[].aggregation), and where it came from
(card::Provenance). Each already has a home.
§Facets
The parameters that decide whether two records are comparable are grouped
into seven facets — model, task, harness, generation, trial, grading, env —
and each facet gets a fingerprint (see evalhub_core::fingerprint). The
rule for what goes into a facet’s core keys, as opposed to ext, is in
facet. Keys that are recorded but must not affect the fingerprint are
marked in the schema with x-fingerprint: false; the code has no separate
exclusion list.
§Closed core, open ext
Every object in the core is closed (additionalProperties: false, via
#[serde(deny_unknown_fields)]). The only open places are the top-level
ext and each facet’s ext, which are maps keyed by namespace
({ns}/{name}) so that two producers’ extensions cannot collide. A hub
that accepted unknown keys silently would let the contract erode one
producer at a time.
§What the client does not send
A record carries no id and no created_at. The record is the client’s
claim; the identifier and the time are the hub’s facts, assigned on
POST and returned in the response. The content_hash (sha256 of the
canonical record) is likewise computed by the hub and is used for
idempotency, never as an address.
§Numbers
Numeric values are JSON numbers, read as IEEE 754 doubles. An integer
outside ±2^53 is rejected (number_too_large); a producer with such a
value sends it as a string. null means recorded but unknown; an absent
key means not recorded. The two are different and both are preserved.
§Modules
card— the Card record: results, counts, provenance, redaction.eval— the Eval header: origin, source kind, default conditions.run— a run of an Eval: status, per-run facets, metrics, its own attachments.facet— the seven facets and their core keys.common— pieces shared by Card, Eval and Run: producer, attachments, relations,ext.query— the query request and response envelope.error— the422 { errors: [...] }envelope and its code enum.openapi— the parts of the OpenAPI document that are not derived from a handler (info, servers, shared error responses).
§Versioning of the schema itself
Each record names its schema ("schema": "evalhub.card/1.1"). A change
that adds an optional key is a minor bump; a change that removes a key,
changes a type, or tightens a constraint is a major bump and a new
schemas/card-2.json alongside the old one. The hub keeps accepting the
previous major for at least one release after the new one ships, and
then refuses it.
A kind can therefore have more than one accepted identifier.
RecordKind::schema_id and RecordKind::schema name the current
one, which is what a producer should write; RecordKind::schema_ids
lists every identifier the hub accepts, and RecordKind::schema_for
returns the document for one of them. A minor bump keeps one document
whose schema key accepts every minor of the major; a major bump adds a
document.
The current state: evalhub.card/1.1 added the optional run_results,
and evalhub.card/1.0 remains accepted under the same document
(schemas/card.json). evalhub.eval/2.0 removed runs from the Eval
body, a major bump, so it has its own document (schemas/eval-2.json,
served as eval-2) and is the only Eval identifier accepted.
The previous Eval major, evalhub.eval/1.0, was accepted by releases
0.2.0 and 0.3.0, which converted such a body at ingest into a 2.0 header
and its runs. Release 0.4.0 closed that window: a body declaring
evalhub.eval/1.0 is refused with 422 runs_moved, the error a 2.0
header carrying runs gets, and the 1.0 type (eval::v1) and its
document (schemas/eval.json, served as eval) were removed. Stored
0.1.x bodies are converted once by the store’s data migration, which
keeps its own reading of the 1.0 shape.
Modules§
- card
- The Card record: what was measured, how, and what the score was.
- common
- Pieces shared by Card, Eval and Run: producer, attachments, relations,
ext. - error
- The error envelope and the closed set of error codes.
- eval
- The Eval record: the material a Card was measured from.
- facet
- The seven facets: the parameters that decide comparability.
- openapi
- The hand-written skeleton of the OpenAPI 3.1 document.
- query
- The query request and response envelope.
- run
- The Run: one execution inside an Eval, as a row of the record.
Enums§
- Record
Kind - The two record kinds, as they appear in URLs (
/cards,/evals) and in thetypecolumn of the store. Everything that differs between a Card and an Eval (its schema, its facets, its semantic rules) is keyed on this.
Constants§
- CARD_
SCHEMA - Schema identifier a Card record should declare: the current one.
evalhub.card/1.0is still accepted (seeRecordKind::schema_ids). - EVAL_
SCHEMA - Schema identifier an Eval header should declare, and the only one the hub accepts for an Eval (see “Versioning of the schema itself”).
- SCHEMA_
NAMES - Names under which the server serves the generated schemas
(
GET /schemas/{name}), in the orderall_schemasreturns them.eval-2is the Eval header;runis one run. There is noeval: that name served theevalhub.eval/1.0document, removed in release 0.4.0.
Functions§
- all_
schemas - Every served schema with its name, in
SCHEMA_NAMESorder. The committed files underschemas/are generated from this list, and a test asserts they are current. - schema_
for_ card - The JSON Schema of a
card::Card.$idis not set; the server sets it to the URL it serves the schema from. - schema_
for_ error - The JSON Schema of the
error::ErrorEnvelopereturned with422/409(and413for the size codes). - schema_
for_ eval - The JSON Schema of an
eval::Evalheader (evalhub.eval/2.0), served aseval-2. - schema_
for_ query - The JSON Schema of the
query::QueryRequestenvelope. The filter insidewhereis left open here; the query language defines it. - schema_
for_ run - The JSON Schema of a
run::Run.