1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
//! The Eval record: the material a Card was measured from.
//!
//! An Eval is a named, versioned set of runs, prompts, tasks or traces. It is
//! what makes two Cards comparable: if both point at the same Eval version
//! through `core/uses_eval`, they were measured on the same material, and the
//! comparison view in `evalhub_server::api::relations` can line them up.
//!
//! The name is deliberately not "Source". Source suggests source code; an
//! Eval is the evaluation itself, as a body of material, distinct from the
//! Card that summarises it.
//!
//! # Shape
//!
//! ```text
//! Eval
//! ├── schema "evalhub.eval/1.0"
//! ├── title
//! ├── producer { name, version }
//! ├── eval_kind run_set | prompt_set | task_set | trace_set
//! ├── origin live | reconstructed | imported
//! ├── model / task / harness / generation / trial / env (no grading facet)
//! ├── runs[] { run_id, started_at, ended_at, outcome, calls, artifacts[] }
//! │ required when eval_kind = run_set
//! ├── relations[] derived_from / subset_of → other Evals
//! ├── attachments[]
//! ├── redaction
//! └── ext
//! ```
//!
//! A `run_set` example:
//!
//! ```json
//! {
//! "schema": "evalhub.eval/1.0",
//! "title": "single2 K4, 33 runs",
//! "producer": {"name": "my-harness", "version": "0.4.1"},
//! "eval_kind": "run_set",
//! "origin": "live",
//! "harness": {"name": "my-harness", "version": "0.4.1"},
//! "model": {"id": "qwen3.6-32b"},
//! "task": {"id": "single2", "version": "3", "split": "test", "n": 33},
//! "generation": {"temperature": 0.0},
//! "trial": {"k": 4},
//! "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
//! "runs": [{
//! "run_id": "r1",
//! "started_at": "2026-09-20T10:00:00Z", "ended_at": "2026-09-20T10:02:11Z",
//! "outcome": "pass",
//! "calls": "calls/r1.jsonl",
//! "artifacts": ["artifacts/r1/diff.patch"]
//! }],
//! "relations": [{"type": "core/subset_of", "to": "alice/single2-full@1"}],
//! "attachments": [
//! {"path": "calls/r1.jsonl", "sha256": "…", "size": 90210, "media_type": "application/x-ndjson"},
//! {"path": "artifacts/r1/diff.patch", "sha256": "…", "size": 3311, "media_type": "text/x-diff"}
//! ],
//! "redaction": {"applied": false},
//! "ext": {}
//! }
//! ```
//!
//! # `eval_kind` and `origin`
//!
//! `eval_kind` says what the material is. A `run_set` has `runs[]`, each with
//! an outcome and a pointer (`calls`) to an attachment holding the model
//! calls. The other kinds carry their material entirely in attachments.
//!
//! `origin` says how the material came to exist: `live` (captured while the
//! evaluation ran), `reconstructed` (rebuilt afterwards from logs),
//! `imported` (converted from another format, such as an Inspect `.eval`
//! file or an EEE datastore entry). A reader comparing Cards can weigh a
//! reconstructed Eval differently; the hub does not.
//!
//! # No grading facet
//!
//! An Eval has six of the seven facets. Grading belongs to the Card, because
//! the same material can be graded several ways, and each grading is a
//! different Card pointing at the same Eval.
//!
//! # Privacy across the relation
//!
//! A public Card may point at a private Eval. A reader without access sees
//! that an Eval exists, its `version_id`, and the `content_hash` commitment —
//! enough to know the Card is anchored to something specific — and nothing
//! else. No title, no runs, no attachment URLs.
//!
//! The Card's own body does not reveal more than its edges. On every read
//! that returns a body, a `relations[]` element whose target the reader may
//! not see is removed and listed under `withheld.relations` as the same
//! commitment (`type`, `version_id`, `content_hash`). The body returned is
//! then not the stored one, and its `content_hash` does not match it. A
//! query's `relations.to` condition matches only targets the caller may
//! see, so a search cannot confirm a private name either.
//!
//! A write learns no more than a read. A target the writer may not see is
//! stored unresolved, exactly as one that does not exist: the response and
//! the `refs_resolved` badge are the same in both cases. The edge resolves
//! for a reader once the target becomes visible to them.
use ;
use JsonSchema;
use ;
use crate;
use crate;
/// An Eval: the material a Card was measured from.
/// What an Eval's material is.
/// How an Eval's material came to exist.
/// One run in a `run_set`.
/// How a run ended.