Skip to main content

evalhub_schema/
lib.rs

1//! Record types for evalhub, and the source of its public contract.
2//!
3//! evalhub is a hosting service for LLM evaluation results. This crate holds
4//! the shape of what the hub accepts and returns: the two record kinds, the
5//! facets they share, the query language's request and response, and the
6//! error envelope. Everything else in the workspace is built on these types.
7//!
8//! # What lives here and what does not
9//!
10//! This crate is **types only**. It has no I/O, no validation beyond what
11//! `serde` enforces structurally, no knowledge of Postgres or HTTP. The
12//! semantic rules (counts must add up, attachment references must resolve,
13//! integers must fit in 2^53) live in `evalhub_core`; the transport lives in
14//! `evalhub_server`. Keeping the types free of behaviour is what lets them be
15//! the contract: a client that deserialises `schemas/card.json` sees exactly
16//! what the server sees.
17//!
18//! # The contract is generated from these types, not the other way round
19//!
20//! The Rust types are the source. Each public record type derives
21//! [`schemars::JsonSchema`], and the JSON Schema (draft 2020-12) is generated
22//! from it and committed under `schemas/`. A snapshot test asserts that the
23//! generated schema equals the committed file, so a change to a type is not
24//! done until the schema in the same commit has changed with it. The OpenAPI
25//! 3.1 document served by `evalhub_server` embeds the same schemas, produced by
26//! the same derive, so there is one description of a Card and it is this
27//! crate.
28//!
29//! The alternative — hand-written JSON Schema with Rust types generated from
30//! it — was rejected: generated Rust cannot carry the `serde` attributes and
31//! doc comments the server needs, and two hand-maintained documents (record
32//! schema and OpenAPI) drift.
33//!
34//! Doc comments on these types are public. They become the `description` in
35//! the served schema and in `openapi.json`, so they are written for the person
36//! writing a client, not for the person reading this crate.
37//!
38//! # The two record kinds
39//!
40//! A `card::Card` says *what was measured, how, and what the score was*. An
41//! `eval::Eval` is *the material*: a set of runs, prompts, tasks or traces
42//! that a Card was measured from. Both are published under a name,
43//! `{ns}/{name}`, as an append-only sequence of immutable versions (the one
44//! exception, the 0.2.0 data migration that rewrote stored
45//! `evalhub.eval/1.0` bodies into 2.0 headers, is `evalhub_store`'s). The
46//! relation between them (`core/uses_eval`) is what makes comparison possible:
47//! two Cards that point at the same Eval version measured the same thing.
48//!
49//! An Eval's versioned body is a *header*. Its runs (`run::Run`) are rows
50//! of the Eval record, outside every version: writing a run does not append
51//! a header version. A run records what happened and what was measured
52//! directly; the verdict on it is the Card's (`card::Card::run_results`).
53//!
54//! The runs a Card judged are its *used set*, per Eval it uses: the
55//! `attrs.runs` of its `core/uses_eval` relations, or, without them, every
56//! run neither archived nor deleted when the Card is posted ([`common`],
57//! [`card`]). Four hashes tie the pieces together, all computed by the
58//! hub and none sent by the client: the header's `content_hash`, each
59//! run's `content_hash`, the record's `runs_hash` over every run, and a
60//! used-set hash over the runs a Card used; the formulas are
61//! `evalhub_core`'s. Runs have no visibility of their own and follow their
62//! Eval; a Card's `run_results` follow the Card, except that a reader who
63//! may not see an Eval is never shown the entries that name it. How many
64//! runs a batch and how many `run_results` a Card may carry are the
65//! server's limits, not constraints of these types.
66//!
67//! A Card has no `kind` field. Whether a score came from a judge, an
68//! aggregation, or was transcribed from a paper is not a kind of Card; those
69//! are properties of how it was graded (`facet::Grading`), how it was
70//! aggregated (`results[].aggregation`), and where it came from
71//! (`card::Provenance`). Each already has a home.
72//!
73//! # Facets
74//!
75//! The parameters that decide whether two records are comparable are grouped
76//! into seven facets — model, task, harness, generation, trial, grading, env —
77//! and each facet gets a fingerprint (see `evalhub_core::fingerprint`). The
78//! rule for what goes into a facet's core keys, as opposed to `ext`, is in
79//! [`facet`]. Keys that are recorded but must not affect the fingerprint are
80//! marked in the schema with `x-fingerprint: false`; the code has no separate
81//! exclusion list.
82//!
83//! # Closed core, open `ext`
84//!
85//! Every object in the core is closed (`additionalProperties: false`, via
86//! `#[serde(deny_unknown_fields)]`). The only open places are the top-level
87//! `ext` and each facet's `ext`, which are maps keyed by namespace
88//! (`{ns}/{name}`) so that two producers' extensions cannot collide. A hub
89//! that accepted unknown keys silently would let the contract erode one
90//! producer at a time.
91//!
92//! # What the client does not send
93//!
94//! A record carries no `id` and no `created_at`. The record is the client's
95//! claim; the identifier and the time are the hub's facts, assigned on
96//! `POST` and returned in the response. The `content_hash` (sha256 of the
97//! canonical record) is likewise computed by the hub and is used for
98//! idempotency, never as an address.
99//!
100//! # Numbers
101//!
102//! Numeric values are JSON numbers, read as IEEE 754 doubles. An integer
103//! outside ±2^53 is rejected (`number_too_large`); a producer with such a
104//! value sends it as a string. `null` means *recorded but unknown*; an absent
105//! key means *not recorded*. The two are different and both are preserved.
106//!
107//! # Modules
108//!
109//! - [`card`] — the Card record: results, counts, provenance, redaction.
110//! - [`eval`] — the Eval header: origin, source kind, default conditions;
111//!   and [`eval::v1`], the 1.0 shape with `runs[]` in the body.
112//! - [`run`] — a run of an Eval: status, per-run facets, metrics, its own
113//!   attachments.
114//! - [`facet`] — the seven facets and their core keys.
115//! - [`common`] — pieces shared by Card, Eval and Run: producer, attachments,
116//!   relations, `ext`.
117//! - [`query`] — the query request and response envelope.
118//! - [`error`] — the `422 { errors: [...] }` envelope and its code enum.
119//! - [`openapi`] — the parts of the OpenAPI document that are not derived
120//!   from a handler (info, servers, shared error responses).
121//!
122//! # Versioning of the schema itself
123//!
124//! Each record names its schema (`"schema": "evalhub.card/1.1"`). A change
125//! that adds an optional key is a minor bump; a change that removes a key,
126//! changes a type, or tightens a constraint is a major bump and a new
127//! `schemas/card-2.json` alongside the old one. The hub keeps accepting the
128//! previous major for at least one release.
129//!
130//! A kind can therefore have more than one accepted identifier.
131//! [`RecordKind::schema_id`] and [`RecordKind::schema`] name the current
132//! one, which is what a producer should write; [`RecordKind::schema_ids`]
133//! lists every identifier the hub accepts, and [`RecordKind::schema_for`]
134//! returns the document for one of them. A minor bump keeps one document
135//! whose `schema` key accepts every minor of the major; a major bump adds a
136//! document.
137//!
138//! The current state: `evalhub.card/1.1` added the optional `run_results`,
139//! and `evalhub.card/1.0` remains accepted under the same document
140//! (`schemas/card.json`). `evalhub.eval/2.0` removed `runs` from the Eval
141//! body, a major bump, so it has its own document (`schemas/eval-2.json`)
142//! beside the 1.0 one (`schemas/eval.json`, from [`eval::v1`]). Release
143//! 0.2.0 is the one release that still accepts `evalhub.eval/1.0` bodies,
144//! converting them at ingest into a 2.0 header and its runs; release 0.3.0
145//! removes `evalhub.eval/1.0`, [`eval::v1`] and `schemas/eval.json`.
146
147pub mod card;
148pub mod common;
149pub mod error;
150pub mod eval;
151pub mod facet;
152pub mod openapi;
153pub mod query;
154pub mod run;
155
156/// Schema identifier a Card record should declare: the current one.
157/// `evalhub.card/1.0` is still accepted (see [`RecordKind::schema_ids`]).
158pub const CARD_SCHEMA: &str = "evalhub.card/1.1";
159
160/// Schema identifier an Eval header should declare: the current one.
161/// `evalhub.eval/1.0` is accepted by release 0.2.0 only (see
162/// [`RecordKind::schema_ids`] and [`eval::v1`]).
163pub const EVAL_SCHEMA: &str = "evalhub.eval/2.0";
164
165/// The previous minor of the Card schema, still accepted, described by the
166/// same document as [`CARD_SCHEMA`].
167const CARD_SCHEMA_1_0: &str = "evalhub.card/1.0";
168
169/// The previous major of the Eval schema, accepted by release 0.2.0 only.
170const EVAL_SCHEMA_1_0: &str = "evalhub.eval/1.0";
171
172/// Names under which the server serves the generated schemas
173/// (`GET /schemas/{name}`), in the order [`all_schemas`] returns them.
174/// `eval` is the 1.0 document (with `runs[]`) until release 0.3.0 removes
175/// it; `eval-2` is the current Eval header; `run` is one run.
176pub const SCHEMA_NAMES: [&str; 6] = ["card", "eval", "eval-2", "run", "error", "query"];
177
178/// The two record kinds, as they appear in URLs (`/cards`, `/evals`) and in
179/// the `type` column of the store. Everything that differs between a Card
180/// and an Eval (its schema, its facets, its semantic rules) is keyed on this.
181#[derive(
182    Debug,
183    Clone,
184    Copy,
185    PartialEq,
186    Eq,
187    Hash,
188    serde::Serialize,
189    serde::Deserialize,
190    schemars::JsonSchema,
191)]
192#[serde(rename_all = "snake_case")]
193pub enum RecordKind {
194    /// A Card: what was measured, how, and what the score was.
195    Card,
196    /// An Eval: the material a Card was measured from.
197    Eval,
198}
199
200impl RecordKind {
201    /// The current `schema` identifier for this kind: the one a producer
202    /// should declare. Other identifiers may still be accepted; see
203    /// [`RecordKind::schema_ids`].
204    pub const fn schema_id(self) -> &'static str {
205        match self {
206            RecordKind::Card => CARD_SCHEMA,
207            RecordKind::Eval => EVAL_SCHEMA,
208        }
209    }
210
211    /// Every `schema` identifier the hub accepts for this kind, the current
212    /// one ([`RecordKind::schema_id`]) first.
213    pub const fn schema_ids(self) -> &'static [&'static str] {
214        match self {
215            RecordKind::Card => &[CARD_SCHEMA, CARD_SCHEMA_1_0],
216            RecordKind::Eval => &[EVAL_SCHEMA, EVAL_SCHEMA_1_0],
217        }
218    }
219
220    /// The generated JSON Schema for the current identifier of this kind.
221    pub fn schema(self) -> schemars::Schema {
222        match self {
223            RecordKind::Card => schema_for_card(),
224            RecordKind::Eval => schema_for_eval(),
225        }
226    }
227
228    /// The generated JSON Schema for one accepted identifier of this kind,
229    /// or `None` when `id` is not one of [`RecordKind::schema_ids`]. Both
230    /// Card identifiers share one document; each Eval major has its own.
231    pub fn schema_for(self, id: &str) -> Option<schemars::Schema> {
232        match (self, id) {
233            (RecordKind::Card, CARD_SCHEMA | CARD_SCHEMA_1_0) => Some(schema_for_card()),
234            (RecordKind::Eval, EVAL_SCHEMA) => Some(schema_for_eval()),
235            (RecordKind::Eval, EVAL_SCHEMA_1_0) => Some(schema_for_eval_v1()),
236            _ => None,
237        }
238    }
239
240    /// The short name: `card` or `eval`.
241    pub const fn as_str(self) -> &'static str {
242        match self {
243            RecordKind::Card => "card",
244            RecordKind::Eval => "eval",
245        }
246    }
247}
248
249/// Generator settings shared by every schema: JSON Schema draft 2020-12, the
250/// dialect OpenAPI 3.1 uses, so the served files and `openapi.json` agree.
251fn generator() -> schemars::SchemaGenerator {
252    schemars::generate::SchemaSettings::draft2020_12().into_generator()
253}
254
255/// The JSON Schema of a [`card::Card`]. `$id` is not set; the server sets it
256/// to the URL it serves the schema from.
257pub fn schema_for_card() -> schemars::Schema {
258    generator().into_root_schema_for::<card::Card>()
259}
260
261/// The JSON Schema of an [`eval::Eval`] header (`evalhub.eval/2.0`), served
262/// as `eval-2`.
263pub fn schema_for_eval() -> schemars::Schema {
264    generator().into_root_schema_for::<eval::Eval>()
265}
266
267/// The JSON Schema of an [`eval::v1::Eval`] (`evalhub.eval/1.0`, with
268/// `runs[]`), served as `eval` until release 0.3.0 removes it.
269pub fn schema_for_eval_v1() -> schemars::Schema {
270    generator().into_root_schema_for::<eval::v1::Eval>()
271}
272
273/// The JSON Schema of a [`run::Run`].
274pub fn schema_for_run() -> schemars::Schema {
275    generator().into_root_schema_for::<run::Run>()
276}
277
278/// The JSON Schema of the [`error::ErrorEnvelope`] returned with `422` / `409`
279/// (and `413` for the size codes).
280pub fn schema_for_error() -> schemars::Schema {
281    generator().into_root_schema_for::<error::ErrorEnvelope>()
282}
283
284/// The JSON Schema of the [`query::QueryRequest`] envelope. The filter inside
285/// `where` is left open here; the query language defines it.
286pub fn schema_for_query() -> schemars::Schema {
287    generator().into_root_schema_for::<query::QueryRequest>()
288}
289
290/// Every served schema with its name, in [`SCHEMA_NAMES`] order. The
291/// committed files under `schemas/` are generated from this list, and a test
292/// asserts they are current.
293pub fn all_schemas() -> Vec<(&'static str, schemars::Schema)> {
294    vec![
295        ("card", schema_for_card()),
296        ("eval", schema_for_eval_v1()),
297        ("eval-2", schema_for_eval()),
298        ("run", schema_for_run()),
299        ("error", schema_for_error()),
300        ("query", schema_for_query()),
301    ]
302}