evalhub_schema/eval.rs
1//! The Eval record: the material a Card was measured from.
2//!
3//! An Eval is a named, versioned header plus the runs that belong to it. It
4//! is what makes two Cards comparable: if both point at the same Eval
5//! version through `core/uses_eval`, they were measured on the same
6//! material, and the comparison view in `evalhub_server::api::relations`
7//! can line them up.
8//!
9//! The name is deliberately not "Source". Source suggests source code; an
10//! Eval is the evaluation itself, as a body of material, distinct from the
11//! Card that summarises it.
12//!
13//! # Shape
14//!
15//! ```text
16//! Eval (record, {ns}/{name})
17//! ├── header this type; one immutable body per version
18//! │ ├── schema "evalhub.eval/2.0"
19//! │ ├── title
20//! │ ├── producer { name, version }
21//! │ ├── eval_kind run_set | prompt_set | task_set | trace_set
22//! │ ├── origin live | reconstructed | imported
23//! │ ├── model / task / harness / generation / trial / env
24//! │ │ defaults for the runs (no grading facet)
25//! │ ├── relations[] derived_from / subset_of → other Evals
26//! │ ├── attachments[]
27//! │ ├── redaction
28//! │ └── ext
29//! └── runs rows of the record, outside every version
30//! Run { run_id, status, error?, skip_reason?, started_at?, ended_at?,
31//! model / task / harness / generation / trial / env,
32//! calls?, artifacts[], attachments[], metrics{}, ext, meta? }
33//! see `crate::run`
34//! ```
35//!
36//! The header is what this type describes and what `POST /evals/{ns}/{name}`
37//! appends as a version. The runs are not in it. They belong to the
38//! *record*, not to a version: a run is written on its own, keyed by
39//! `(record, run_id)`, and adding or overwriting one does not append a
40//! header version. A header version is appended when the description or the
41//! default conditions change. This is the change from `evalhub.eval/1.0`,
42//! whose body carried `runs[]`. The hub accepted that shape in releases
43//! 0.2.0 and 0.3.0 and refuses it since 0.4.0 (`422 runs_moved`); the 1.0
44//! type is no longer part of this crate.
45//!
46//! A header example:
47//!
48//! ```json
49//! {
50//! "schema": "evalhub.eval/2.0",
51//! "title": "single2 K4, 33 runs",
52//! "producer": {"name": "my-harness", "version": "0.4.1"},
53//! "eval_kind": "run_set",
54//! "origin": "live",
55//! "harness": {"name": "my-harness", "version": "0.4.1"},
56//! "model": {"id": "qwen3.6-32b"},
57//! "task": {"id": "single2", "version": "3", "split": "test", "n": 33},
58//! "generation": {"temperature": 0.0},
59//! "trial": {"k": 4},
60//! "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
61//! "relations": [{"type": "core/subset_of", "to": "alice/single2-full@1"}],
62//! "attachments": [
63//! {"path": "tasks.jsonl", "sha256": "…", "size": 90210, "media_type": "application/x-ndjson"}
64//! ],
65//! "redaction": {"applied": false},
66//! "ext": {}
67//! }
68//! ```
69//!
70//! # `eval_kind` and `origin`
71//!
72//! `eval_kind` says what the material is. It no longer decides whether the
73//! Eval has runs: a header is complete with zero runs (a `run_set` header
74//! is typically posted first and its runs written afterwards), and an Eval
75//! of any `eval_kind` may have runs. The 1.0 rule "`runs` is required when
76//! `eval_kind` is `run_set`" is gone.
77//!
78//! `origin` says how the material came to exist: `live` (captured while the
79//! evaluation ran), `reconstructed` (rebuilt afterwards from logs),
80//! `imported` (converted from another format, such as an Inspect `.eval`
81//! file or an EEE datastore entry). A reader comparing Cards can weigh a
82//! reconstructed Eval differently; the hub does not.
83//!
84//! # Conditions are per run; the header's facets are defaults
85//!
86//! One session may change the model, the temperature or the harness
87//! between runs, so the conditions a run was measured under belong to the
88//! run: [`crate::run::Run`] has the same six facets as the header. The
89//! header's facets are *defaults*. A run that omits a facet is given the
90//! header's value at the time it is written, and the copy is stored with
91//! the run; changing the header's defaults later does not change the
92//! conditions of runs already written. A facet the run carries itself wins
93//! over the default. Fingerprints are computed per run from the run's own
94//! facets; the header's fingerprint remains, as the fingerprint of the
95//! defaults.
96//!
97//! # Tasks and runs are not separate types
98//!
99//! There is no sample or task type beside the run. A run's `calls` log
100//! carries the inputs (the prompt, the reference) along with the outputs,
101//! which is enough material. A set of tasks with nothing executed yet is an
102//! Eval whose runs have empty outputs, and the link between a task Eval and
103//! an execution Eval is an ordinary relation between Evals
104//! (`core/derived_from`, `core/subset_of`). Repeated attempts at the same
105//! task (`trial.k`, a rerun after an error) are separate runs; overwriting a
106//! run is for correcting that same attempt. A producer that wants to prove
107//! two runs saw the same input puts a hash of it in the run's `ext`.
108//!
109//! # A run has no verdict
110//!
111//! A run records what happened ([`crate::run::RunStatus`]: `ok`, `error`,
112//! `skipped`) and what was measured directly ([`crate::run::Run::metrics`]).
113//! It never records whether it passed. Pass and fail depend on a grader and
114//! a threshold, and the same material can be graded several ways, so the
115//! verdict lives on the Card that graded it, in
116//! [`crate::card::Card::run_results`].
117//!
118//! # No grading facet
119//!
120//! An Eval (header and runs alike) has six of the seven facets. Grading
121//! belongs to the Card, because the same material can be graded several
122//! ways, and each grading is a different Card pointing at the same Eval.
123//!
124//! # Privacy across the relation
125//!
126//! A public Card may point at a private Eval. A reader without access sees
127//! that an Eval exists, its `version_id`, and the `content_hash` commitment —
128//! enough to know the Card is anchored to something specific — and nothing
129//! else. No title, no runs, no attachment URLs.
130//!
131//! The Card's own body does not reveal more than its edges. On every read
132//! that returns a body, a `relations[]` element whose target the reader may
133//! not see is removed and listed under `withheld.relations` as the same
134//! commitment (`type`, `version_id`, `content_hash`). The body returned is
135//! then not the stored one, and its `content_hash` does not match it. A
136//! query's `relations.to` condition matches only targets the caller may
137//! see, so a search cannot confirm a private name either.
138//!
139//! A write learns no more than a read. A target the writer may not see is
140//! stored unresolved, exactly as one that does not exist: the response and
141//! the `refs_resolved` badge are the same in both cases. The edge resolves
142//! for a reader once the target becomes visible to them.
143
144use schemars::JsonSchema;
145use serde::{Deserialize, Serialize};
146
147use crate::common::{Attachment, Ext, Producer, Redaction, Relation};
148use crate::facet::{Env, Generation, Harness, Model, Task, Trial};
149
150/// An Eval header: what the material is, how it came to exist, and the
151/// default conditions for its runs. The runs themselves are not part of the
152/// body; they belong to the record and are written one by one
153/// (`schemas/run.json`).
154#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
155#[serde(deny_unknown_fields)]
156pub struct Eval {
157 /// Schema identifier; must be `evalhub.eval/2.0`.
158 #[schemars(extend("const" = "evalhub.eval/2.0"))]
159 pub schema: String,
160 /// Human-readable title.
161 pub title: String,
162 /// The software that wrote this record.
163 pub producer: Producer,
164 /// What the material is.
165 pub eval_kind: EvalKind,
166 /// How the material came to exist.
167 pub origin: Origin,
168 /// The model facet: the default for runs that do not carry their own.
169 #[serde(default, skip_serializing_if = "Option::is_none")]
170 pub model: Option<Model>,
171 /// The task facet: the default for runs that do not carry their own.
172 #[serde(default, skip_serializing_if = "Option::is_none")]
173 pub task: Option<Task>,
174 /// The harness facet: the default for runs that do not carry their own.
175 #[serde(default, skip_serializing_if = "Option::is_none")]
176 pub harness: Option<Harness>,
177 /// The generation facet: the default for runs that do not carry their own.
178 #[serde(default, skip_serializing_if = "Option::is_none")]
179 pub generation: Option<Generation>,
180 /// The trial facet: the default for runs that do not carry their own.
181 #[serde(default, skip_serializing_if = "Option::is_none")]
182 pub trial: Option<Trial>,
183 /// The environment facet: the default for runs that do not carry their own.
184 #[serde(default, skip_serializing_if = "Option::is_none")]
185 pub env: Option<Env>,
186 /// Edges to other Evals (`core/derived_from`, `core/subset_of`).
187 #[serde(default, skip_serializing_if = "Vec::is_empty")]
188 pub relations: Vec<Relation>,
189 /// Files this header refers to. A run's own files are in the run's
190 /// `attachments[]`, not here.
191 #[serde(default, skip_serializing_if = "Vec::is_empty")]
192 pub attachments: Vec<Attachment>,
193 /// What the producer removed before publishing.
194 #[serde(default, skip_serializing_if = "Option::is_none")]
195 pub redaction: Option<Redaction>,
196 /// Producer-private extensions keyed by namespace.
197 #[serde(default)]
198 pub ext: Ext,
199}
200
201/// What an Eval's material is.
202#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
203#[serde(rename_all = "snake_case")]
204pub enum EvalKind {
205 /// A set of runs, each with its model calls.
206 RunSet,
207 /// A set of prompts.
208 PromptSet,
209 /// A set of tasks.
210 TaskSet,
211 /// A set of traces.
212 TraceSet,
213}
214
215/// How an Eval's material came to exist.
216#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
217#[serde(rename_all = "snake_case")]
218pub enum Origin {
219 /// Captured while the evaluation ran.
220 Live,
221 /// Rebuilt afterwards from logs.
222 Reconstructed,
223 /// Converted from another format.
224 Imported,
225}