evalhub_schema/eval.rs
1//! The Eval record: the material a Card was measured from.
2//!
3//! An Eval is a named, versioned header plus the runs that belong to it. It
4//! is what makes two Cards comparable: if both point at the same Eval
5//! version through `core/uses_eval`, they were measured on the same
6//! material, and the comparison view in `evalhub_server::api::relations`
7//! can line them up.
8//!
9//! The name is deliberately not "Source". Source suggests source code; an
10//! Eval is the evaluation itself, as a body of material, distinct from the
11//! Card that summarises it.
12//!
13//! # Shape
14//!
15//! ```text
16//! Eval (record, {ns}/{name})
17//! ├── header this type; one immutable body per version
18//! │ ├── schema "evalhub.eval/2.0"
19//! │ ├── title
20//! │ ├── producer { name, version }
21//! │ ├── eval_kind run_set | prompt_set | task_set | trace_set
22//! │ ├── origin live | reconstructed | imported
23//! │ ├── model / task / harness / generation / trial / env
24//! │ │ defaults for the runs (no grading facet)
25//! │ ├── relations[] derived_from / subset_of → other Evals
26//! │ ├── attachments[]
27//! │ ├── redaction
28//! │ └── ext
29//! └── runs rows of the record, outside every version
30//! Run { run_id, status, error?, skip_reason?, started_at?, ended_at?,
31//! model / task / harness / generation / trial / env,
32//! calls?, artifacts[], attachments[], metrics{}, ext, meta? }
33//! see `crate::run`
34//! ```
35//!
36//! The header is what this type describes and what `POST /evals/{ns}/{name}`
37//! appends as a version. The runs are not in it. They belong to the
38//! *record*, not to a version: a run is written on its own, keyed by
39//! `(record, run_id)`, and adding or overwriting one does not append a
40//! header version. A header version is appended when the description or the
41//! default conditions change. This is the change from `evalhub.eval/1.0`,
42//! whose body carried `runs[]` (the 1.0 shape is kept in [`v1`] for the one
43//! release that still accepts it).
44//!
45//! A header example:
46//!
47//! ```json
48//! {
49//! "schema": "evalhub.eval/2.0",
50//! "title": "single2 K4, 33 runs",
51//! "producer": {"name": "my-harness", "version": "0.4.1"},
52//! "eval_kind": "run_set",
53//! "origin": "live",
54//! "harness": {"name": "my-harness", "version": "0.4.1"},
55//! "model": {"id": "qwen3.6-32b"},
56//! "task": {"id": "single2", "version": "3", "split": "test", "n": 33},
57//! "generation": {"temperature": 0.0},
58//! "trial": {"k": 4},
59//! "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
60//! "relations": [{"type": "core/subset_of", "to": "alice/single2-full@1"}],
61//! "attachments": [
62//! {"path": "tasks.jsonl", "sha256": "…", "size": 90210, "media_type": "application/x-ndjson"}
63//! ],
64//! "redaction": {"applied": false},
65//! "ext": {}
66//! }
67//! ```
68//!
69//! # `eval_kind` and `origin`
70//!
71//! `eval_kind` says what the material is. It no longer decides whether the
72//! Eval has runs: a header is complete with zero runs (a `run_set` header
73//! is typically posted first and its runs written afterwards), and an Eval
74//! of any `eval_kind` may have runs. The 1.0 rule "`runs` is required when
75//! `eval_kind` is `run_set`" is gone.
76//!
77//! `origin` says how the material came to exist: `live` (captured while the
78//! evaluation ran), `reconstructed` (rebuilt afterwards from logs),
79//! `imported` (converted from another format, such as an Inspect `.eval`
80//! file or an EEE datastore entry). A reader comparing Cards can weigh a
81//! reconstructed Eval differently; the hub does not.
82//!
83//! # Conditions are per run; the header's facets are defaults
84//!
85//! One session may change the model, the temperature or the harness
86//! between runs, so the conditions a run was measured under belong to the
87//! run: [`crate::run::Run`] has the same six facets as the header. The
88//! header's facets are *defaults*. A run that omits a facet is given the
89//! header's value at the time it is written, and the copy is stored with
90//! the run; changing the header's defaults later does not change the
91//! conditions of runs already written. A facet the run carries itself wins
92//! over the default. Fingerprints are computed per run from the run's own
93//! facets; the header's fingerprint remains, as the fingerprint of the
94//! defaults.
95//!
96//! # Tasks and runs are not separate types
97//!
98//! There is no sample or task type beside the run. A run's `calls` log
99//! carries the inputs (the prompt, the reference) along with the outputs,
100//! which is enough material. A set of tasks with nothing executed yet is an
101//! Eval whose runs have empty outputs, and the link between a task Eval and
102//! an execution Eval is an ordinary relation between Evals
103//! (`core/derived_from`, `core/subset_of`). Repeated attempts at the same
104//! task (`trial.k`, a rerun after an error) are separate runs; overwriting a
105//! run is for correcting that same attempt. A producer that wants to prove
106//! two runs saw the same input puts a hash of it in the run's `ext`.
107//!
108//! # A run has no verdict
109//!
110//! A run records what happened ([`crate::run::RunStatus`]: `ok`, `error`,
111//! `skipped`) and what was measured directly ([`crate::run::Run::metrics`]).
112//! It never records whether it passed. Pass and fail depend on a grader and
113//! a threshold, and the same material can be graded several ways, so the
114//! verdict lives on the Card that graded it, in
115//! [`crate::card::Card::run_results`].
116//!
117//! # No grading facet
118//!
119//! An Eval (header and runs alike) has six of the seven facets. Grading
120//! belongs to the Card, because the same material can be graded several
121//! ways, and each grading is a different Card pointing at the same Eval.
122//!
123//! # Privacy across the relation
124//!
125//! A public Card may point at a private Eval. A reader without access sees
126//! that an Eval exists, its `version_id`, and the `content_hash` commitment —
127//! enough to know the Card is anchored to something specific — and nothing
128//! else. No title, no runs, no attachment URLs.
129//!
130//! The Card's own body does not reveal more than its edges. On every read
131//! that returns a body, a `relations[]` element whose target the reader may
132//! not see is removed and listed under `withheld.relations` as the same
133//! commitment (`type`, `version_id`, `content_hash`). The body returned is
134//! then not the stored one, and its `content_hash` does not match it. A
135//! query's `relations.to` condition matches only targets the caller may
136//! see, so a search cannot confirm a private name either.
137//!
138//! A write learns no more than a read. A target the writer may not see is
139//! stored unresolved, exactly as one that does not exist: the response and
140//! the `refs_resolved` badge are the same in both cases. The edge resolves
141//! for a reader once the target becomes visible to them.
142
143use schemars::JsonSchema;
144use serde::{Deserialize, Serialize};
145
146use crate::common::{Attachment, Ext, Producer, Redaction, Relation};
147use crate::facet::{Env, Generation, Harness, Model, Task, Trial};
148
149pub mod v1;
150
151/// An Eval header: what the material is, how it came to exist, and the
152/// default conditions for its runs. The runs themselves are not part of the
153/// body; they belong to the record and are written one by one
154/// (`schemas/run.json`).
155#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
156#[serde(deny_unknown_fields)]
157pub struct Eval {
158 /// Schema identifier; must be `evalhub.eval/2.0`.
159 #[schemars(extend("const" = "evalhub.eval/2.0"))]
160 pub schema: String,
161 /// Human-readable title.
162 pub title: String,
163 /// The software that wrote this record.
164 pub producer: Producer,
165 /// What the material is.
166 pub eval_kind: EvalKind,
167 /// How the material came to exist.
168 pub origin: Origin,
169 /// The model facet: the default for runs that do not carry their own.
170 #[serde(default, skip_serializing_if = "Option::is_none")]
171 pub model: Option<Model>,
172 /// The task facet: the default for runs that do not carry their own.
173 #[serde(default, skip_serializing_if = "Option::is_none")]
174 pub task: Option<Task>,
175 /// The harness facet: the default for runs that do not carry their own.
176 #[serde(default, skip_serializing_if = "Option::is_none")]
177 pub harness: Option<Harness>,
178 /// The generation facet: the default for runs that do not carry their own.
179 #[serde(default, skip_serializing_if = "Option::is_none")]
180 pub generation: Option<Generation>,
181 /// The trial facet: the default for runs that do not carry their own.
182 #[serde(default, skip_serializing_if = "Option::is_none")]
183 pub trial: Option<Trial>,
184 /// The environment facet: the default for runs that do not carry their own.
185 #[serde(default, skip_serializing_if = "Option::is_none")]
186 pub env: Option<Env>,
187 /// Edges to other Evals (`core/derived_from`, `core/subset_of`).
188 #[serde(default, skip_serializing_if = "Vec::is_empty")]
189 pub relations: Vec<Relation>,
190 /// Files this header refers to. A run's own files are in the run's
191 /// `attachments[]`, not here.
192 #[serde(default, skip_serializing_if = "Vec::is_empty")]
193 pub attachments: Vec<Attachment>,
194 /// What the producer removed before publishing.
195 #[serde(default, skip_serializing_if = "Option::is_none")]
196 pub redaction: Option<Redaction>,
197 /// Producer-private extensions keyed by namespace.
198 #[serde(default)]
199 pub ext: Ext,
200}
201
202/// What an Eval's material is.
203#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
204#[serde(rename_all = "snake_case")]
205pub enum EvalKind {
206 /// A set of runs, each with its model calls.
207 RunSet,
208 /// A set of prompts.
209 PromptSet,
210 /// A set of tasks.
211 TaskSet,
212 /// A set of traces.
213 TraceSet,
214}
215
216/// How an Eval's material came to exist.
217#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
218#[serde(rename_all = "snake_case")]
219pub enum Origin {
220 /// Captured while the evaluation ran.
221 Live,
222 /// Rebuilt afterwards from logs.
223 Reconstructed,
224 /// Converted from another format.
225 Imported,
226}