1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
//! Record types for evalhub, and the source of its public contract.
//!
//! evalhub is a hosting service for LLM evaluation results. This crate holds
//! the shape of what the hub accepts and returns: the two record kinds, the
//! facets they share, the query language's request and response, and the
//! error envelope. Everything else in the workspace is built on these types.
//!
//! # What lives here and what does not
//!
//! This crate is **types only**. It has no I/O, no validation beyond what
//! `serde` enforces structurally, no knowledge of Postgres or HTTP. The
//! semantic rules (counts must add up, attachment references must resolve,
//! integers must fit in 2^53) live in `evalhub_core`; the transport lives in
//! `evalhub_server`. Keeping the types free of behaviour is what lets them be
//! the contract: a client that deserialises `schemas/card.json` sees exactly
//! what the server sees.
//!
//! # The contract is generated from these types, not the other way round
//!
//! The Rust types are the source. Each public record type derives
//! [`schemars::JsonSchema`], and the JSON Schema (draft 2020-12) is generated
//! from it and committed under `schemas/`. A snapshot test asserts that the
//! generated schema equals the committed file, so a change to a type is not
//! done until the schema in the same commit has changed with it. The OpenAPI
//! 3.1 document served by `evalhub_server` embeds the same schemas, produced by
//! the same derive, so there is one description of a Card and it is this
//! crate.
//!
//! The alternative — hand-written JSON Schema with Rust types generated from
//! it — was rejected: generated Rust cannot carry the `serde` attributes and
//! doc comments the server needs, and two hand-maintained documents (record
//! schema and OpenAPI) drift.
//!
//! Doc comments on these types are public. They become the `description` in
//! the served schema and in `openapi.json`, so they are written for the person
//! writing a client, not for the person reading this crate.
//!
//! # The two record kinds
//!
//! A `card::Card` says *what was measured, how, and what the score was*. An
//! `eval::Eval` is *the material*: a set of runs, prompts, tasks or traces
//! that a Card was measured from. Both are published under a name,
//! `{ns}/{name}`, as an append-only sequence of immutable versions (the one
//! exception, the 0.2.0 data migration that rewrote stored
//! `evalhub.eval/1.0` bodies into 2.0 headers, is `evalhub_store`'s). The
//! relation between them (`core/uses_eval`) is what makes comparison possible:
//! two Cards that point at the same Eval version measured the same thing.
//!
//! An Eval's versioned body is a *header*. Its runs (`run::Run`) are rows
//! of the Eval record, outside every version: writing a run does not append
//! a header version. A run records what happened and what was measured
//! directly; the verdict on it is the Card's (`card::Card::run_results`).
//!
//! The runs a Card judged are its *used set*, per Eval it uses: the
//! `attrs.runs` of its `core/uses_eval` relations, or, without them, every
//! run neither archived nor deleted when the Card is posted ([`common`],
//! [`card`]). Four hashes tie the pieces together, all computed by the
//! hub and none sent by the client: the header's `content_hash`, each
//! run's `content_hash`, the record's `runs_hash` over every run, and a
//! used-set hash over the runs a Card used; the formulas are
//! `evalhub_core`'s. Runs have no visibility of their own and follow their
//! Eval; a Card's `run_results` follow the Card, except that a reader who
//! may not see an Eval is never shown the entries that name it. How many
//! runs a batch and how many `run_results` a Card may carry are the
//! server's limits, not constraints of these types.
//!
//! A Card has no `kind` field. Whether a score came from a judge, an
//! aggregation, or was transcribed from a paper is not a kind of Card; those
//! are properties of how it was graded (`facet::Grading`), how it was
//! aggregated (`results[].aggregation`), and where it came from
//! (`card::Provenance`). Each already has a home.
//!
//! # Facets
//!
//! The parameters that decide whether two records are comparable are grouped
//! into seven facets — model, task, harness, generation, trial, grading, env —
//! and each facet gets a fingerprint (see `evalhub_core::fingerprint`). The
//! rule for what goes into a facet's core keys, as opposed to `ext`, is in
//! [`facet`]. Keys that are recorded but must not affect the fingerprint are
//! marked in the schema with `x-fingerprint: false`; the code has no separate
//! exclusion list.
//!
//! # Closed core, open `ext`
//!
//! Every object in the core is closed (`additionalProperties: false`, via
//! `#[serde(deny_unknown_fields)]`). The only open places are the top-level
//! `ext` and each facet's `ext`, which are maps keyed by namespace
//! (`{ns}/{name}`) so that two producers' extensions cannot collide. A hub
//! that accepted unknown keys silently would let the contract erode one
//! producer at a time.
//!
//! # What the client does not send
//!
//! A record carries no `id` and no `created_at`. The record is the client's
//! claim; the identifier and the time are the hub's facts, assigned on
//! `POST` and returned in the response. The `content_hash` (sha256 of the
//! canonical record) is likewise computed by the hub and is used for
//! idempotency, never as an address.
//!
//! # Numbers
//!
//! Numeric values are JSON numbers, read as IEEE 754 doubles. An integer
//! outside ±2^53 is rejected (`number_too_large`); a producer with such a
//! value sends it as a string. `null` means *recorded but unknown*; an absent
//! key means *not recorded*. The two are different and both are preserved.
//!
//! # Modules
//!
//! - [`card`] — the Card record: results, counts, provenance, redaction.
//! - [`eval`] — the Eval header: origin, source kind, default conditions.
//! - [`run`] — a run of an Eval: status, per-run facets, metrics, its own
//! attachments.
//! - [`facet`] — the seven facets and their core keys.
//! - [`common`] — pieces shared by Card, Eval and Run: producer, attachments,
//! relations, `ext`.
//! - [`query`] — the query request and response envelope.
//! - [`error`] — the `422 { errors: [...] }` envelope and its code enum.
//! - [`openapi`] — the parts of the OpenAPI document that are not derived
//! from a handler (info, servers, shared error responses).
//!
//! # Versioning of the schema itself
//!
//! Each record names its schema (`"schema": "evalhub.card/1.1"`). A change
//! that adds an optional key is a minor bump; a change that removes a key,
//! changes a type, or tightens a constraint is a major bump and a new
//! `schemas/card-2.json` alongside the old one. The hub keeps accepting the
//! previous major for at least one release after the new one ships, and
//! then refuses it.
//!
//! A kind can therefore have more than one accepted identifier.
//! [`RecordKind::schema_id`] and [`RecordKind::schema`] name the current
//! one, which is what a producer should write; [`RecordKind::schema_ids`]
//! lists every identifier the hub accepts, and [`RecordKind::schema_for`]
//! returns the document for one of them. A minor bump keeps one document
//! whose `schema` key accepts every minor of the major; a major bump adds a
//! document.
//!
//! The current state: `evalhub.card/1.1` added the optional `run_results`,
//! and `evalhub.card/1.0` remains accepted under the same document
//! (`schemas/card.json`). `evalhub.eval/2.0` removed `runs` from the Eval
//! body, a major bump, so it has its own document (`schemas/eval-2.json`,
//! served as `eval-2`) and is the only Eval identifier accepted.
//!
//! The previous Eval major, `evalhub.eval/1.0`, was accepted by releases
//! 0.2.0 and 0.3.0, which converted such a body at ingest into a 2.0 header
//! and its runs. Release 0.4.0 closed that window: a body declaring
//! `evalhub.eval/1.0` is refused with `422 runs_moved`, the error a 2.0
//! header carrying `runs` gets, and the 1.0 type (`eval::v1`) and its
//! document (`schemas/eval.json`, served as `eval`) were removed. Stored
//! 0.1.x bodies are converted once by the store's data migration, which
//! keeps its own reading of the 1.0 shape.
/// Schema identifier a Card record should declare: the current one.
/// `evalhub.card/1.0` is still accepted (see [`RecordKind::schema_ids`]).
pub const CARD_SCHEMA: &str = "evalhub.card/1.1";
/// Schema identifier an Eval header should declare, and the only one the
/// hub accepts for an Eval (see "Versioning of the schema itself").
pub const EVAL_SCHEMA: &str = "evalhub.eval/2.0";
/// The previous minor of the Card schema, still accepted, described by the
/// same document as [`CARD_SCHEMA`].
const CARD_SCHEMA_1_0: &str = "evalhub.card/1.0";
/// Names under which the server serves the generated schemas
/// (`GET /schemas/{name}`), in the order [`all_schemas`] returns them.
/// `eval-2` is the Eval header; `run` is one run. There is no `eval`: that
/// name served the `evalhub.eval/1.0` document, removed in release 0.4.0.
pub const SCHEMA_NAMES: = ;
/// The two record kinds, as they appear in URLs (`/cards`, `/evals`) and in
/// the `type` column of the store. Everything that differs between a Card
/// and an Eval (its schema, its facets, its semantic rules) is keyed on this.
/// Generator settings shared by every schema: JSON Schema draft 2020-12, the
/// dialect OpenAPI 3.1 uses, so the served files and `openapi.json` agree.
/// The JSON Schema of a [`card::Card`]. `$id` is not set; the server sets it
/// to the URL it serves the schema from.
/// The JSON Schema of an [`eval::Eval`] header (`evalhub.eval/2.0`), served
/// as `eval-2`.
/// The JSON Schema of a [`run::Run`].
/// The JSON Schema of the [`error::ErrorEnvelope`] returned with `422` / `409`
/// (and `413` for the size codes).
/// The JSON Schema of the [`query::QueryRequest`] envelope. The filter inside
/// `where` is left open here; the query language defines it.
/// Every served schema with its name, in [`SCHEMA_NAMES`] order. The
/// committed files under `schemas/` are generated from this list, and a test
/// asserts they are current.