evalhub_schema/lib.rs
1//! Record types for evalhub, and the source of its public contract.
2//!
3//! evalhub is a hosting service for LLM evaluation results. This crate holds
4//! the shape of what the hub accepts and returns: the two record kinds, the
5//! facets they share, the query language's request and response, and the
6//! error envelope. Everything else in the workspace is built on these types.
7//!
8//! # What lives here and what does not
9//!
10//! This crate is **types only**. It has no I/O, no validation beyond what
11//! `serde` enforces structurally, no knowledge of Postgres or HTTP. The
12//! semantic rules (counts must add up, attachment references must resolve,
13//! integers must fit in 2^53) live in `evalhub_core`; the transport lives in
14//! `evalhub_server`. Keeping the types free of behaviour is what lets them be
15//! the contract: a client that deserialises `schemas/card.json` sees exactly
16//! what the server sees.
17//!
18//! # The contract is generated from these types, not the other way round
19//!
20//! The Rust types are the source. Each public record type derives
21//! [`schemars::JsonSchema`], and the JSON Schema (draft 2020-12) is generated
22//! from it and committed under `schemas/`. A snapshot test asserts that the
23//! generated schema equals the committed file, so a change to a type is not
24//! done until the schema in the same commit has changed with it. The OpenAPI
25//! 3.1 document served by `evalhub_server` embeds the same schemas, produced by
26//! the same derive, so there is one description of a Card and it is this
27//! crate.
28//!
29//! The alternative — hand-written JSON Schema with Rust types generated from
30//! it — was rejected: generated Rust cannot carry the `serde` attributes and
31//! doc comments the server needs, and two hand-maintained documents (record
32//! schema and OpenAPI) drift.
33//!
34//! Doc comments on these types are public. They become the `description` in
35//! the served schema and in `openapi.json`, so they are written for the person
36//! writing a client, not for the person reading this crate.
37//!
38//! # The two record kinds
39//!
40//! A `card::Card` says *what was measured, how, and what the score was*. An
41//! `eval::Eval` is *the material*: a set of runs, prompts, tasks or traces
42//! that a Card was measured from. Both are published under a name,
43//! `{ns}/{name}`, as an append-only sequence of immutable versions. The
44//! relation between them (`core/uses_eval`) is what makes comparison possible:
45//! two Cards that point at the same Eval version measured the same thing.
46//!
47//! A Card has no `kind` field. Whether a score came from a judge, an
48//! aggregation, or was transcribed from a paper is not a kind of Card; those
49//! are properties of how it was graded (`facet::Grading`), how it was
50//! aggregated (`results[].aggregation`), and where it came from
51//! (`card::Provenance`). Each already has a home.
52//!
53//! # Facets
54//!
55//! The parameters that decide whether two records are comparable are grouped
56//! into seven facets — model, task, harness, generation, trial, grading, env —
57//! and each facet gets a fingerprint (see `evalhub_core::fingerprint`). The
58//! rule for what goes into a facet's core keys, as opposed to `ext`, is in
59//! [`facet`]. Keys that are recorded but must not affect the fingerprint are
60//! marked in the schema with `x-fingerprint: false`; the code has no separate
61//! exclusion list.
62//!
63//! # Closed core, open `ext`
64//!
65//! Every object in the core is closed (`additionalProperties: false`, via
66//! `#[serde(deny_unknown_fields)]`). The only open places are the top-level
67//! `ext` and each facet's `ext`, which are maps keyed by namespace
68//! (`{ns}/{name}`) so that two producers' extensions cannot collide. A hub
69//! that accepted unknown keys silently would let the contract erode one
70//! producer at a time.
71//!
72//! # What the client does not send
73//!
74//! A record carries no `id` and no `created_at`. The record is the client's
75//! claim; the identifier and the time are the hub's facts, assigned on
76//! `POST` and returned in the response. The `content_hash` (sha256 of the
77//! canonical record) is likewise computed by the hub and is used for
78//! idempotency, never as an address.
79//!
80//! # Numbers
81//!
82//! Numeric values are JSON numbers, read as IEEE 754 doubles. An integer
83//! outside ±2^53 is rejected (`number_too_large`); a producer with such a
84//! value sends it as a string. `null` means *recorded but unknown*; an absent
85//! key means *not recorded*. The two are different and both are preserved.
86//!
87//! # Modules
88//!
89//! - [`card`] — the Card record: results, counts, provenance, redaction.
90//! - [`eval`] — the Eval record: runs, origin, source kind.
91//! - [`facet`] — the seven facets and their core keys.
92//! - [`common`] — pieces shared by both records: producer, attachments,
93//! relations, `ext`.
94//! - [`query`] — the query request and response envelope.
95//! - [`error`] — the `422 { errors: [...] }` envelope and its code enum.
96//! - [`openapi`] — the parts of the OpenAPI document that are not derived
97//! from a handler (info, servers, shared error responses).
98//!
99//! # Versioning of the schema itself
100//!
101//! Each record names its schema (`"schema": "evalhub.card/1.0"`). A change
102//! that adds an optional key is a minor bump; a change that removes a key,
103//! changes a type, or tightens a constraint is a major bump and a new
104//! `schemas/card-2.json` alongside the old one. The hub keeps accepting the
105//! previous major for at least one release.
106
107pub mod card;
108pub mod common;
109pub mod error;
110pub mod eval;
111pub mod facet;
112pub mod openapi;
113pub mod query;
114
115/// Schema identifier carried by every Card record.
116pub const CARD_SCHEMA: &str = "evalhub.card/1.0";
117
118/// Schema identifier carried by every Eval record.
119pub const EVAL_SCHEMA: &str = "evalhub.eval/1.0";
120
121/// Names under which the server serves the generated schemas
122/// (`GET /schemas/{name}`), in the order [`all_schemas`] returns them.
123pub const SCHEMA_NAMES: [&str; 4] = ["card", "eval", "error", "query"];
124
125/// The two record kinds, as they appear in URLs (`/cards`, `/evals`) and in
126/// the `type` column of the store. Everything that differs between a Card
127/// and an Eval (its schema, its facets, its semantic rules) is keyed on this.
128#[derive(
129 Debug,
130 Clone,
131 Copy,
132 PartialEq,
133 Eq,
134 Hash,
135 serde::Serialize,
136 serde::Deserialize,
137 schemars::JsonSchema,
138)]
139#[serde(rename_all = "snake_case")]
140pub enum RecordKind {
141 /// A Card: what was measured, how, and what the score was.
142 Card,
143 /// An Eval: the material a Card was measured from.
144 Eval,
145}
146
147impl RecordKind {
148 /// The `schema` identifier a record of this kind must declare.
149 pub const fn schema_id(self) -> &'static str {
150 match self {
151 RecordKind::Card => CARD_SCHEMA,
152 RecordKind::Eval => EVAL_SCHEMA,
153 }
154 }
155
156 /// The generated JSON Schema for this kind.
157 pub fn schema(self) -> schemars::Schema {
158 match self {
159 RecordKind::Card => schema_for_card(),
160 RecordKind::Eval => schema_for_eval(),
161 }
162 }
163
164 /// The short name: `card` or `eval`.
165 pub const fn as_str(self) -> &'static str {
166 match self {
167 RecordKind::Card => "card",
168 RecordKind::Eval => "eval",
169 }
170 }
171}
172
173/// Generator settings shared by every schema: JSON Schema draft 2020-12, the
174/// dialect OpenAPI 3.1 uses, so the served files and `openapi.json` agree.
175fn generator() -> schemars::SchemaGenerator {
176 schemars::generate::SchemaSettings::draft2020_12().into_generator()
177}
178
179/// The JSON Schema of a [`card::Card`]. `$id` is not set; the server sets it
180/// to the URL it serves the schema from.
181pub fn schema_for_card() -> schemars::Schema {
182 generator().into_root_schema_for::<card::Card>()
183}
184
185/// The JSON Schema of an [`eval::Eval`].
186pub fn schema_for_eval() -> schemars::Schema {
187 generator().into_root_schema_for::<eval::Eval>()
188}
189
190/// The JSON Schema of the [`error::ErrorEnvelope`] returned with `422` / `409`.
191pub fn schema_for_error() -> schemars::Schema {
192 generator().into_root_schema_for::<error::ErrorEnvelope>()
193}
194
195/// The JSON Schema of the [`query::QueryRequest`] envelope. The filter inside
196/// `where` is left open here; the query language defines it.
197pub fn schema_for_query() -> schemars::Schema {
198 generator().into_root_schema_for::<query::QueryRequest>()
199}
200
201/// Every served schema with its name, in [`SCHEMA_NAMES`] order. The
202/// committed files under `schemas/` are generated from this list, and a test
203/// asserts they are current.
204pub fn all_schemas() -> Vec<(&'static str, schemars::Schema)> {
205 vec![
206 ("card", schema_for_card()),
207 ("eval", schema_for_eval()),
208 ("error", schema_for_error()),
209 ("query", schema_for_query()),
210 ]
211}