1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
//! The Card record: what was measured, how, and what the score was.
//!
//! A Card is the unit a harness publishes after an evaluation. It is a claim
//! made by the producer, kept verbatim by the hub. The hub adds an id, a
//! sequence number, a content hash, badges and fingerprints around it, but
//! never edits the record itself; a correction is a new version.
//!
//! # Shape
//!
//! ```text
//! Card
//! ├── schema "evalhub.card/1.1" (or "evalhub.card/1.0")
//! ├── title free text
//! ├── producer { name, version } who wrote this record
//! ├── model / task / harness / generation / trial / grading / env
//! │ seven facets (see `facet`) what was measured, under what conditions
//! ├── results[] { metric, value, n, aggregation, uncertainty, by, samples_ref }
//! ├── counts { attempted, completed, failed, skipped, errored }
//! ├── run_results[] { eval, run_id, metric, value?, label?, by?, from? }
//! │ per-run verdicts on the runs of a cited Eval (1.1)
//! ├── relations[] { type, to, attrs } edges to other records
//! ├── attachments[] { path, sha256, size, media_type }
//! ├── redaction { applied, method, fields }
//! ├── provenance { source_type, evaluator_relationship }
//! └── ext { "{ns}/{name}": { ... } } producer-private extension
//! ```
//!
//! A complete example:
//!
//! ```json
//! {
//! "schema": "evalhub.card/1.1",
//! "title": "qwen3.6-32b on single2 K4",
//! "producer": {"name": "my-harness", "version": "0.4.1"},
//! "model": {"id": "qwen3.6-32b", "provider": "local", "quantization": {"method": "awq", "dtype": "int4"}},
//! "task": {"id": "single2", "version": "3", "split": "test", "n": 33, "seed": 7},
//! "harness": {"name": "my-harness", "version": "0.4.1", "prompt_template_sha256": "…"},
//! "generation": {"temperature": 0.0, "max_tokens": 4096},
//! "trial": {"k": 4, "pass_at_k": true, "max_turns": 20, "sandbox": {"image": "ghcr.io/x/y", "digest": "sha256:…"}},
//! "grading": {"graders": [{"name": "exact_match", "kind": "deterministic"}]},
//! "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
//! "results": [{
//! "metric": "core/pass_rate",
//! "value": 0.62, "n": 33,
//! "aggregation": "mean",
//! "uncertainty": {"stderr": 0.08, "ci": {"level": 0.95, "low": 0.45, "high": 0.77}},
//! "by": {"grader": "exact_match"},
//! "samples_ref": "samples.jsonl"
//! }],
//! "counts": {"attempted": 33, "completed": 30, "failed": 2, "skipped": 1, "errored": 0},
//! "relations": [{"type": "core/uses_eval", "to": "alice/single2-k4@3"}],
//! "attachments": [{"path": "samples.jsonl", "sha256": "…", "size": 12345, "media_type": "application/x-ndjson"}],
//! "redaction": {"applied": true, "method": "regex+llm", "fields": ["samples.jsonl:response"]},
//! "provenance": {"source_type": "evaluation_run", "evaluator_relationship": "first_party"},
//! "ext": {"alice/qwen-loop": {"rung": 4}}
//! }
//! ```
//!
//! # Results
//!
//! `results[].metric` is a registry id in the form `{ns}/{name}`
//! (`core/pass_rate`). A metric the registry does not know is still accepted
//! if it has that form; a bare name is rejected (`metric_id_invalid`). Whether
//! the metric is registered is surfaced as the `metric_registered` badge, not
//! as an error — the hub records claims, it does not gate on vocabulary.
//!
//! `aggregation` names how `value` was derived from the samples (`mean`,
//! `pass_at_k`, `median`, `sum`, `custom`). `uncertainty` carries whatever the
//! producer computed (standard error, a confidence interval). `by` records the
//! grader or other partition the number applies to. `samples_ref` names an
//! entry in `attachments[]` holding the per-sample rows; the hub checks that
//! the path exists in `attachments[]` and nothing else about it.
//!
//! # Counts
//!
//! If `results` is non-empty, `counts` is required and must satisfy
//! `attempted >= completed + failed + skipped + errored`. This is the one
//! arithmetic rule the hub enforces, because a score without the denominator
//! it was computed over is not comparable to anything.
//!
//! # Run results
//!
//! ```text
//! relations[core/uses_eval] { to: "{ns}/{name}@{seq}", attrs: { runs?: [run_id] } }
//! run_results[] { eval: "{ns}/{name}", run_id, metric, value?, label?, by?, from? }
//! ```
//!
//! `results[]` is the aggregate; `run_results[]` is the verdict per run
//! that the aggregate was computed from. A run of an Eval records what
//! happened and what was measured directly (`crate::run`); it never says
//! whether it passed. That judgement depends on a grader, so it is the
//! Card's claim and lives here: one element per `(run, metric)` the Card
//! judged, with a number (`value`), a label (`label`, such as `pass` or
//! `fail`), or both. At least one of the two is present
//! (`run_result_value_or_label`). `by` partitions the verdict the way
//! `results[].by` does, and `from` optionally names the run's `metrics`
//! keys the verdict was derived from. The hub does not recompute anything
//! and does not check that `results[]` agrees with `run_results[]`; how the
//! verdicts were produced is the grading facet's to describe.
//!
//! Re-grading appends a Card version; the earlier verdicts stay with the
//! earlier version. Grading the same runs with a different grader is a
//! different Card.
//!
//! `run_results` was added in `evalhub.card/1.1`. It is an optional key, so
//! the bump is minor: one schema document serves both identifiers, a Card
//! declaring `evalhub.card/1.0` may carry `run_results` too, and Cards
//! stored under 1.0 are not rewritten.
//!
//! ## The used set
//!
//! Which runs a Card used is decided per cited Eval record, and that set of
//! `run_id`s is the Card's *used set* for the record. For each Eval record,
//! it is the union of `attrs.runs` over the Card's `core/uses_eval`
//! relations that resolve to that record; or, when none of those relations
//! carries `attrs.runs`, every run of the record that is neither archived
//! nor deleted at the moment the Card is posted. The hub fixes the used set
//! when the Card is posted and remembers each run's content hash with it,
//! which is how a later change to a run is detected.
//!
//! Against the used set:
//!
//! - `run_results[].eval` names the record (`{ns}/{name}`, no `@seq`) of
//! one of the Card's `core/uses_eval` relations
//! (`run_results_eval_unknown`).
//! - every `run_id` in a used set has a row in that Eval, archived and
//! deleted rows included (`run_unknown`). A run is always in the hub, so
//! a missing row is a mistake, not an unresolved reference, and an Eval
//! the writer may not see answers the same way as a run that does not
//! exist.
//! - every `run_results[].run_id` is in the used set of its `eval`
//! (`run_not_in_used_set`).
//!
//! # Provenance and redaction
//!
//! `provenance` says where the numbers came from (`evaluation_run`,
//! `paper`, `model_card`, …) and who ran it relative to the model
//! (`first_party`, `third_party`). `redaction` says whether the producer
//! removed anything before publishing, by what method, and from which fields.
//! The hub does not redact; it only makes the producer's statement visible
//! and searchable, and awards the `redacted` badge when `applied` is true.
//!
//! # Fields the hub owns
//!
//! `id`, `version_id`, `seq`, `label`, `content_hash`, `created_at`,
//! `changed[]`, `badges[]` and the per-facet fingerprints are not part of
//! this type. They are returned alongside it in API responses and are
//! documented in `evalhub_server::api::records`.
use JsonSchema;
use ;
use crate;
use crate;
/// A Card: what was measured, how, and what the score was.
/// One score.
/// The Card's verdict on one run for one metric.
/// How a score was derived from its samples.
/// Uncertainty of a score.
/// A confidence interval.
/// Item counts. `attempted >= completed + failed + skipped + errored`.
/// Where the numbers came from.
/// The kind of source a Card's numbers were taken from.
/// Who ran the evaluation relative to the model's maker.