Skip to main content

evalhub_schema/
eval.rs

1//! The Eval record: the material a Card was measured from.
2//!
3//! An Eval is a named, versioned header plus the runs that belong to it. It
4//! is what makes two Cards comparable: if both point at the same Eval
5//! version through `core/uses_eval`, they were measured on the same
6//! material, and the comparison view in `evalhub_server::api::relations`
7//! can line them up.
8//!
9//! The name is deliberately not "Source". Source suggests source code; an
10//! Eval is the evaluation itself, as a body of material, distinct from the
11//! Card that summarises it.
12//!
13//! # Shape
14//!
15//! ```text
16//! Eval (record, {ns}/{name})
17//! ├── header        this type; one immutable body per version
18//! │   ├── schema        "evalhub.eval/2.0"
19//! │   ├── title
20//! │   ├── producer      { name, version }
21//! │   ├── eval_kind     run_set | prompt_set | task_set | trace_set
22//! │   ├── origin        live | reconstructed | imported
23//! │   ├── model / task / harness / generation / trial / env
24//! │   │                 defaults for the runs (no grading facet)
25//! │   ├── relations[]   derived_from / subset_of → other Evals
26//! │   ├── attachments[]
27//! │   ├── redaction
28//! │   └── ext
29//! └── runs          rows of the record, outside every version
30//!     Run { run_id, status, error?, skip_reason?, started_at?, ended_at?,
31//!           model / task / harness / generation / trial / env,
32//!           calls?, artifacts[], attachments[], metrics{}, ext, meta? }
33//!                   see `crate::run`
34//! ```
35//!
36//! The header is what this type describes and what `POST /evals/{ns}/{name}`
37//! appends as a version. The runs are not in it. They belong to the
38//! *record*, not to a version: a run is written on its own, keyed by
39//! `(record, run_id)`, and adding or overwriting one does not append a
40//! header version. A header version is appended when the description or the
41//! default conditions change. This is the change from `evalhub.eval/1.0`,
42//! whose body carried `runs[]`. The hub accepted that shape in releases
43//! 0.2.0 and 0.3.0 and refuses it since 0.4.0 (`422 runs_moved`); the 1.0
44//! type is no longer part of this crate.
45//!
46//! A header example:
47//!
48//! ```json
49//! {
50//!   "schema": "evalhub.eval/2.0",
51//!   "title": "single2 K4, 33 runs",
52//!   "producer": {"name": "my-harness", "version": "0.4.1"},
53//!   "eval_kind": "run_set",
54//!   "origin": "live",
55//!   "harness": {"name": "my-harness", "version": "0.4.1"},
56//!   "model": {"id": "qwen3.6-32b"},
57//!   "task": {"id": "single2", "version": "3", "split": "test", "n": 33},
58//!   "generation": {"temperature": 0.0},
59//!   "trial": {"k": 4},
60//!   "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
61//!   "relations": [{"type": "core/subset_of", "to": "alice/single2-full@1"}],
62//!   "attachments": [
63//!     {"path": "tasks.jsonl", "sha256": "…", "size": 90210, "media_type": "application/x-ndjson"}
64//!   ],
65//!   "redaction": {"applied": false},
66//!   "ext": {}
67//! }
68//! ```
69//!
70//! # `eval_kind` and `origin`
71//!
72//! `eval_kind` says what the material is. It no longer decides whether the
73//! Eval has runs: a header is complete with zero runs (a `run_set` header
74//! is typically posted first and its runs written afterwards), and an Eval
75//! of any `eval_kind` may have runs. The 1.0 rule "`runs` is required when
76//! `eval_kind` is `run_set`" is gone.
77//!
78//! `origin` says how the material came to exist: `live` (captured while the
79//! evaluation ran), `reconstructed` (rebuilt afterwards from logs),
80//! `imported` (converted from another format, such as an Inspect `.eval`
81//! file or an EEE datastore entry). A reader comparing Cards can weigh a
82//! reconstructed Eval differently; the hub does not.
83//!
84//! # Conditions are per run; the header's facets are defaults
85//!
86//! One session may change the model, the temperature or the harness
87//! between runs, so the conditions a run was measured under belong to the
88//! run: [`crate::run::Run`] has the same six facets as the header. The
89//! header's facets are *defaults*. A run that omits a facet is given the
90//! header's value at the time it is written, and the copy is stored with
91//! the run; changing the header's defaults later does not change the
92//! conditions of runs already written. A facet the run carries itself wins
93//! over the default. Fingerprints are computed per run from the run's own
94//! facets; the header's fingerprint remains, as the fingerprint of the
95//! defaults.
96//!
97//! # Tasks and runs are not separate types
98//!
99//! There is no sample or task type beside the run. A run's `calls` log
100//! carries the inputs (the prompt, the reference) along with the outputs,
101//! which is enough material. A set of tasks with nothing executed yet is an
102//! Eval whose runs have empty outputs, and the link between a task Eval and
103//! an execution Eval is an ordinary relation between Evals
104//! (`core/derived_from`, `core/subset_of`). Repeated attempts at the same
105//! task (`trial.k`, a rerun after an error) are separate runs; overwriting a
106//! run is for correcting that same attempt. A producer that wants to prove
107//! two runs saw the same input puts a hash of it in the run's `ext`.
108//!
109//! # A run has no verdict
110//!
111//! A run records what happened ([`crate::run::RunStatus`]: `ok`, `error`,
112//! `skipped`) and what was measured directly ([`crate::run::Run::metrics`]).
113//! It never records whether it passed. Pass and fail depend on a grader and
114//! a threshold, and the same material can be graded several ways, so the
115//! verdict lives on the Card that graded it, in
116//! [`crate::card::Card::run_results`].
117//!
118//! # No grading facet
119//!
120//! An Eval (header and runs alike) has six of the seven facets. Grading
121//! belongs to the Card, because the same material can be graded several
122//! ways, and each grading is a different Card pointing at the same Eval.
123//!
124//! # Privacy across the relation
125//!
126//! A public Card may point at a private Eval. A reader without access sees
127//! that an Eval exists, its `version_id`, and the `content_hash` commitment —
128//! enough to know the Card is anchored to something specific — and nothing
129//! else. No title, no runs, no attachment URLs.
130//!
131//! The Card's own body does not reveal more than its edges. On every read
132//! that returns a body, a `relations[]` element whose target the reader may
133//! not see is removed and listed under `withheld.relations` as the same
134//! commitment (`type`, `version_id`, `content_hash`). The body returned is
135//! then not the stored one, and its `content_hash` does not match it. A
136//! query's `relations.to` condition matches only targets the caller may
137//! see, so a search cannot confirm a private name either.
138//!
139//! A write learns no more than a read. A target the writer may not see is
140//! stored unresolved, exactly as one that does not exist: the response and
141//! the `refs_resolved` badge are the same in both cases. The edge resolves
142//! for a reader once the target becomes visible to them.
143
144use schemars::JsonSchema;
145use serde::{Deserialize, Serialize};
146
147use crate::common::{Attachment, Ext, Producer, Redaction, Relation};
148use crate::facet::{Env, Generation, Harness, Model, Task, Trial};
149
150/// An Eval header: what the material is, how it came to exist, and the
151/// default conditions for its runs. The runs themselves are not part of the
152/// body; they belong to the record and are written one by one
153/// (`schemas/run.json`).
154#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
155#[serde(deny_unknown_fields)]
156pub struct Eval {
157    /// Schema identifier; must be `evalhub.eval/2.0`.
158    #[schemars(extend("const" = "evalhub.eval/2.0"))]
159    pub schema: String,
160    /// Human-readable title.
161    pub title: String,
162    /// The software that wrote this record.
163    pub producer: Producer,
164    /// What the material is.
165    pub eval_kind: EvalKind,
166    /// How the material came to exist.
167    pub origin: Origin,
168    /// The model facet: the default for runs that do not carry their own.
169    #[serde(default, skip_serializing_if = "Option::is_none")]
170    pub model: Option<Model>,
171    /// The task facet: the default for runs that do not carry their own.
172    #[serde(default, skip_serializing_if = "Option::is_none")]
173    pub task: Option<Task>,
174    /// The harness facet: the default for runs that do not carry their own.
175    #[serde(default, skip_serializing_if = "Option::is_none")]
176    pub harness: Option<Harness>,
177    /// The generation facet: the default for runs that do not carry their own.
178    #[serde(default, skip_serializing_if = "Option::is_none")]
179    pub generation: Option<Generation>,
180    /// The trial facet: the default for runs that do not carry their own.
181    #[serde(default, skip_serializing_if = "Option::is_none")]
182    pub trial: Option<Trial>,
183    /// The environment facet: the default for runs that do not carry their own.
184    #[serde(default, skip_serializing_if = "Option::is_none")]
185    pub env: Option<Env>,
186    /// Edges to other Evals (`core/derived_from`, `core/subset_of`).
187    #[serde(default, skip_serializing_if = "Vec::is_empty")]
188    pub relations: Vec<Relation>,
189    /// Files this header refers to. A run's own files are in the run's
190    /// `attachments[]`, not here.
191    #[serde(default, skip_serializing_if = "Vec::is_empty")]
192    pub attachments: Vec<Attachment>,
193    /// What the producer removed before publishing.
194    #[serde(default, skip_serializing_if = "Option::is_none")]
195    pub redaction: Option<Redaction>,
196    /// Producer-private extensions keyed by namespace.
197    #[serde(default)]
198    pub ext: Ext,
199}
200
201/// What an Eval's material is.
202#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
203#[serde(rename_all = "snake_case")]
204pub enum EvalKind {
205    /// A set of runs, each with its model calls.
206    RunSet,
207    /// A set of prompts.
208    PromptSet,
209    /// A set of tasks.
210    TaskSet,
211    /// A set of traces.
212    TraceSet,
213}
214
215/// How an Eval's material came to exist.
216#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
217#[serde(rename_all = "snake_case")]
218pub enum Origin {
219    /// Captured while the evaluation ran.
220    Live,
221    /// Rebuilt afterwards from logs.
222    Reconstructed,
223    /// Converted from another format.
224    Imported,
225}