evalhub_schema/lib.rs
1//! Record types for evalhub, and the source of its public contract.
2//!
3//! evalhub is a hosting service for LLM evaluation results. This crate holds
4//! the shape of what the hub accepts and returns: the two record kinds, the
5//! facets they share, the query language's request and response, and the
6//! error envelope. Everything else in the workspace is built on these types.
7//!
8//! # What lives here and what does not
9//!
10//! This crate is **types only**. It has no I/O, no validation beyond what
11//! `serde` enforces structurally, no knowledge of Postgres or HTTP. The
12//! semantic rules (counts must add up, attachment references must resolve,
13//! integers must fit in 2^53) live in `evalhub_core`; the transport lives in
14//! `evalhub_server`. Keeping the types free of behaviour is what lets them be
15//! the contract: a client that deserialises `schemas/card.json` sees exactly
16//! what the server sees.
17//!
18//! # The contract is generated from these types, not the other way round
19//!
20//! The Rust types are the source. Each public record type derives
21//! [`schemars::JsonSchema`], and the JSON Schema (draft 2020-12) is generated
22//! from it and committed under `schemas/`. A snapshot test asserts that the
23//! generated schema equals the committed file, so a change to a type is not
24//! done until the schema in the same commit has changed with it. The OpenAPI
25//! 3.1 document served by `evalhub_server` embeds the same schemas, produced by
26//! the same derive, so there is one description of a Card and it is this
27//! crate.
28//!
29//! The alternative — hand-written JSON Schema with Rust types generated from
30//! it — was rejected: generated Rust cannot carry the `serde` attributes and
31//! doc comments the server needs, and two hand-maintained documents (record
32//! schema and OpenAPI) drift.
33//!
34//! Doc comments on these types are public. They become the `description` in
35//! the served schema and in `openapi.json`, so they are written for the person
36//! writing a client, not for the person reading this crate.
37//!
38//! # The two record kinds
39//!
40//! A `card::Card` says *what was measured, how, and what the score was*. An
41//! `eval::Eval` is *the material*: a set of runs, prompts, tasks or traces
42//! that a Card was measured from. Both are published under a name,
43//! `{ns}/{name}`, as an append-only sequence of immutable versions (the one
44//! exception, the 0.2.0 data migration that rewrote stored
45//! `evalhub.eval/1.0` bodies into 2.0 headers, is `evalhub_store`'s). The
46//! relation between them (`core/uses_eval`) is what makes comparison possible:
47//! two Cards that point at the same Eval version measured the same thing.
48//!
49//! An Eval's versioned body is a *header*. Its runs (`run::Run`) are rows
50//! of the Eval record, outside every version: writing a run does not append
51//! a header version. A run records what happened and what was measured
52//! directly; the verdict on it is the Card's (`card::Card::run_results`).
53//!
54//! The runs a Card judged are its *used set*, per Eval it uses: the
55//! `attrs.runs` of its `core/uses_eval` relations, or, without them, every
56//! run neither archived nor deleted when the Card is posted ([`common`],
57//! [`card`]). Four hashes tie the pieces together, all computed by the
58//! hub and none sent by the client: the header's `content_hash`, each
59//! run's `content_hash`, the record's `runs_hash` over every run, and a
60//! used-set hash over the runs a Card used; the formulas are
61//! `evalhub_core`'s. Runs have no visibility of their own and follow their
62//! Eval; a Card's `run_results` follow the Card, except that a reader who
63//! may not see an Eval is never shown the entries that name it. How many
64//! runs a batch and how many `run_results` a Card may carry are the
65//! server's limits, not constraints of these types.
66//!
67//! A Card has no `kind` field. Whether a score came from a judge, an
68//! aggregation, or was transcribed from a paper is not a kind of Card; those
69//! are properties of how it was graded (`facet::Grading`), how it was
70//! aggregated (`results[].aggregation`), and where it came from
71//! (`card::Provenance`). Each already has a home.
72//!
73//! # Facets
74//!
75//! The parameters that decide whether two records are comparable are grouped
76//! into seven facets — model, task, harness, generation, trial, grading, env —
77//! and each facet gets a fingerprint (see `evalhub_core::fingerprint`). The
78//! rule for what goes into a facet's core keys, as opposed to `ext`, is in
79//! [`facet`]. Keys that are recorded but must not affect the fingerprint are
80//! marked in the schema with `x-fingerprint: false`; the code has no separate
81//! exclusion list.
82//!
83//! # Closed core, open `ext`
84//!
85//! Every object in the core is closed (`additionalProperties: false`, via
86//! `#[serde(deny_unknown_fields)]`). The only open places are the top-level
87//! `ext` and each facet's `ext`, which are maps keyed by namespace
88//! (`{ns}/{name}`) so that two producers' extensions cannot collide. A hub
89//! that accepted unknown keys silently would let the contract erode one
90//! producer at a time.
91//!
92//! # What the client does not send
93//!
94//! A record carries no `id` and no `created_at`. The record is the client's
95//! claim; the identifier and the time are the hub's facts, assigned on
96//! `POST` and returned in the response. The `content_hash` (sha256 of the
97//! canonical record) is likewise computed by the hub and is used for
98//! idempotency, never as an address.
99//!
100//! # Numbers
101//!
102//! Numeric values are JSON numbers, read as IEEE 754 doubles. An integer
103//! outside ±2^53 is rejected (`number_too_large`); a producer with such a
104//! value sends it as a string. `null` means *recorded but unknown*; an absent
105//! key means *not recorded*. The two are different and both are preserved.
106//!
107//! # Modules
108//!
109//! - [`card`] — the Card record: results, counts, provenance, redaction.
110//! - [`eval`] — the Eval header: origin, source kind, default conditions.
111//! - [`run`] — a run of an Eval: status, per-run facets, metrics, its own
112//! attachments.
113//! - [`facet`] — the seven facets and their core keys.
114//! - [`common`] — pieces shared by Card, Eval and Run: producer, attachments,
115//! relations, `ext`.
116//! - [`query`] — the query request and response envelope.
117//! - [`error`] — the `422 { errors: [...] }` envelope and its code enum.
118//! - [`openapi`] — the parts of the OpenAPI document that are not derived
119//! from a handler (info, servers, shared error responses).
120//!
121//! # Versioning of the schema itself
122//!
123//! Each record names its schema (`"schema": "evalhub.card/1.1"`). A change
124//! that adds an optional key is a minor bump; a change that removes a key,
125//! changes a type, or tightens a constraint is a major bump and a new
126//! `schemas/card-2.json` alongside the old one. The hub keeps accepting the
127//! previous major for at least one release after the new one ships, and
128//! then refuses it.
129//!
130//! A kind can therefore have more than one accepted identifier.
131//! [`RecordKind::schema_id`] and [`RecordKind::schema`] name the current
132//! one, which is what a producer should write; [`RecordKind::schema_ids`]
133//! lists every identifier the hub accepts, and [`RecordKind::schema_for`]
134//! returns the document for one of them. A minor bump keeps one document
135//! whose `schema` key accepts every minor of the major; a major bump adds a
136//! document.
137//!
138//! The current state: `evalhub.card/1.1` added the optional `run_results`,
139//! and `evalhub.card/1.0` remains accepted under the same document
140//! (`schemas/card.json`). `evalhub.eval/2.0` removed `runs` from the Eval
141//! body, a major bump, so it has its own document (`schemas/eval-2.json`,
142//! served as `eval-2`) and is the only Eval identifier accepted.
143//!
144//! The previous Eval major, `evalhub.eval/1.0`, was accepted by releases
145//! 0.2.0 and 0.3.0, which converted such a body at ingest into a 2.0 header
146//! and its runs. Release 0.4.0 closed that window: a body declaring
147//! `evalhub.eval/1.0` is refused with `422 runs_moved`, the error a 2.0
148//! header carrying `runs` gets, and the 1.0 type (`eval::v1`) and its
149//! document (`schemas/eval.json`, served as `eval`) were removed. Stored
150//! 0.1.x bodies are converted once by the store's data migration, which
151//! keeps its own reading of the 1.0 shape.
152
153pub mod card;
154pub mod common;
155pub mod error;
156pub mod eval;
157pub mod facet;
158pub mod openapi;
159pub mod query;
160pub mod run;
161
162/// Schema identifier a Card record should declare: the current one.
163/// `evalhub.card/1.0` is still accepted (see [`RecordKind::schema_ids`]).
164pub const CARD_SCHEMA: &str = "evalhub.card/1.1";
165
166/// Schema identifier an Eval header should declare, and the only one the
167/// hub accepts for an Eval (see "Versioning of the schema itself").
168pub const EVAL_SCHEMA: &str = "evalhub.eval/2.0";
169
170/// The previous minor of the Card schema, still accepted, described by the
171/// same document as [`CARD_SCHEMA`].
172const CARD_SCHEMA_1_0: &str = "evalhub.card/1.0";
173
174/// Names under which the server serves the generated schemas
175/// (`GET /schemas/{name}`), in the order [`all_schemas`] returns them.
176/// `eval-2` is the Eval header; `run` is one run. There is no `eval`: that
177/// name served the `evalhub.eval/1.0` document, removed in release 0.4.0.
178pub const SCHEMA_NAMES: [&str; 5] = ["card", "eval-2", "run", "error", "query"];
179
180/// The two record kinds, as they appear in URLs (`/cards`, `/evals`) and in
181/// the `type` column of the store. Everything that differs between a Card
182/// and an Eval (its schema, its facets, its semantic rules) is keyed on this.
183#[derive(
184 Debug,
185 Clone,
186 Copy,
187 PartialEq,
188 Eq,
189 Hash,
190 serde::Serialize,
191 serde::Deserialize,
192 schemars::JsonSchema,
193)]
194#[serde(rename_all = "snake_case")]
195pub enum RecordKind {
196 /// A Card: what was measured, how, and what the score was.
197 Card,
198 /// An Eval: the material a Card was measured from.
199 Eval,
200}
201
202impl RecordKind {
203 /// The current `schema` identifier for this kind: the one a producer
204 /// should declare. Other identifiers may still be accepted; see
205 /// [`RecordKind::schema_ids`].
206 pub const fn schema_id(self) -> &'static str {
207 match self {
208 RecordKind::Card => CARD_SCHEMA,
209 RecordKind::Eval => EVAL_SCHEMA,
210 }
211 }
212
213 /// Every `schema` identifier the hub accepts for this kind, the current
214 /// one ([`RecordKind::schema_id`]) first.
215 pub const fn schema_ids(self) -> &'static [&'static str] {
216 match self {
217 RecordKind::Card => &[CARD_SCHEMA, CARD_SCHEMA_1_0],
218 RecordKind::Eval => &[EVAL_SCHEMA],
219 }
220 }
221
222 /// The generated JSON Schema for the current identifier of this kind.
223 pub fn schema(self) -> schemars::Schema {
224 match self {
225 RecordKind::Card => schema_for_card(),
226 RecordKind::Eval => schema_for_eval(),
227 }
228 }
229
230 /// The generated JSON Schema for one accepted identifier of this kind,
231 /// or `None` when `id` is not one of [`RecordKind::schema_ids`]. Both
232 /// Card identifiers share one document.
233 pub fn schema_for(self, id: &str) -> Option<schemars::Schema> {
234 match (self, id) {
235 (RecordKind::Card, CARD_SCHEMA | CARD_SCHEMA_1_0) => Some(schema_for_card()),
236 (RecordKind::Eval, EVAL_SCHEMA) => Some(schema_for_eval()),
237 _ => None,
238 }
239 }
240
241 /// The short name: `card` or `eval`.
242 pub const fn as_str(self) -> &'static str {
243 match self {
244 RecordKind::Card => "card",
245 RecordKind::Eval => "eval",
246 }
247 }
248}
249
250/// Generator settings shared by every schema: JSON Schema draft 2020-12, the
251/// dialect OpenAPI 3.1 uses, so the served files and `openapi.json` agree.
252fn generator() -> schemars::SchemaGenerator {
253 schemars::generate::SchemaSettings::draft2020_12().into_generator()
254}
255
256/// The JSON Schema of a [`card::Card`]. `$id` is not set; the server sets it
257/// to the URL it serves the schema from.
258pub fn schema_for_card() -> schemars::Schema {
259 generator().into_root_schema_for::<card::Card>()
260}
261
262/// The JSON Schema of an [`eval::Eval`] header (`evalhub.eval/2.0`), served
263/// as `eval-2`.
264pub fn schema_for_eval() -> schemars::Schema {
265 generator().into_root_schema_for::<eval::Eval>()
266}
267
268/// The JSON Schema of a [`run::Run`].
269pub fn schema_for_run() -> schemars::Schema {
270 generator().into_root_schema_for::<run::Run>()
271}
272
273/// The JSON Schema of the [`error::ErrorEnvelope`] returned with `422` / `409`
274/// (and `413` for the size codes).
275pub fn schema_for_error() -> schemars::Schema {
276 generator().into_root_schema_for::<error::ErrorEnvelope>()
277}
278
279/// The JSON Schema of the [`query::QueryRequest`] envelope. The filter inside
280/// `where` is left open here; the query language defines it.
281pub fn schema_for_query() -> schemars::Schema {
282 generator().into_root_schema_for::<query::QueryRequest>()
283}
284
285/// Every served schema with its name, in [`SCHEMA_NAMES`] order. The
286/// committed files under `schemas/` are generated from this list, and a test
287/// asserts they are current.
288pub fn all_schemas() -> Vec<(&'static str, schemars::Schema)> {
289 vec![
290 ("card", schema_for_card()),
291 ("eval-2", schema_for_eval()),
292 ("run", schema_for_run()),
293 ("error", schema_for_error()),
294 ("query", schema_for_query()),
295 ]
296}