Skip to main content

evalhub_schema/
eval.rs

1//! The Eval record: the material a Card was measured from.
2//!
3//! An Eval is a named, versioned header plus the runs that belong to it. It
4//! is what makes two Cards comparable: if both point at the same Eval
5//! version through `core/uses_eval`, they were measured on the same
6//! material, and the comparison view in `evalhub_server::api::relations`
7//! can line them up.
8//!
9//! The name is deliberately not "Source". Source suggests source code; an
10//! Eval is the evaluation itself, as a body of material, distinct from the
11//! Card that summarises it.
12//!
13//! # Shape
14//!
15//! ```text
16//! Eval (record, {ns}/{name})
17//! ├── header        this type; one immutable body per version
18//! │   ├── schema        "evalhub.eval/2.0"
19//! │   ├── title
20//! │   ├── producer      { name, version }
21//! │   ├── eval_kind     run_set | prompt_set | task_set | trace_set
22//! │   ├── origin        live | reconstructed | imported
23//! │   ├── model / task / harness / generation / trial / env
24//! │   │                 defaults for the runs (no grading facet)
25//! │   ├── relations[]   derived_from / subset_of → other Evals
26//! │   ├── attachments[]
27//! │   ├── redaction
28//! │   └── ext
29//! └── runs          rows of the record, outside every version
30//!     Run { run_id, status, error?, skip_reason?, started_at?, ended_at?,
31//!           model / task / harness / generation / trial / env,
32//!           calls?, artifacts[], attachments[], metrics{}, ext, meta? }
33//!                   see `crate::run`
34//! ```
35//!
36//! The header is what this type describes and what `POST /evals/{ns}/{name}`
37//! appends as a version. The runs are not in it. They belong to the
38//! *record*, not to a version: a run is written on its own, keyed by
39//! `(record, run_id)`, and adding or overwriting one does not append a
40//! header version. A header version is appended when the description or the
41//! default conditions change. This is the change from `evalhub.eval/1.0`,
42//! whose body carried `runs[]` (the 1.0 shape is kept in [`v1`] for the one
43//! release that still accepts it).
44//!
45//! A header example:
46//!
47//! ```json
48//! {
49//!   "schema": "evalhub.eval/2.0",
50//!   "title": "single2 K4, 33 runs",
51//!   "producer": {"name": "my-harness", "version": "0.4.1"},
52//!   "eval_kind": "run_set",
53//!   "origin": "live",
54//!   "harness": {"name": "my-harness", "version": "0.4.1"},
55//!   "model": {"id": "qwen3.6-32b"},
56//!   "task": {"id": "single2", "version": "3", "split": "test", "n": 33},
57//!   "generation": {"temperature": 0.0},
58//!   "trial": {"k": 4},
59//!   "env": {"git": {"origin": "https://github.com/x/y", "commit": "abc123", "dirty": false}},
60//!   "relations": [{"type": "core/subset_of", "to": "alice/single2-full@1"}],
61//!   "attachments": [
62//!     {"path": "tasks.jsonl", "sha256": "…", "size": 90210, "media_type": "application/x-ndjson"}
63//!   ],
64//!   "redaction": {"applied": false},
65//!   "ext": {}
66//! }
67//! ```
68//!
69//! # `eval_kind` and `origin`
70//!
71//! `eval_kind` says what the material is. It no longer decides whether the
72//! Eval has runs: a header is complete with zero runs (a `run_set` header
73//! is typically posted first and its runs written afterwards), and an Eval
74//! of any `eval_kind` may have runs. The 1.0 rule "`runs` is required when
75//! `eval_kind` is `run_set`" is gone.
76//!
77//! `origin` says how the material came to exist: `live` (captured while the
78//! evaluation ran), `reconstructed` (rebuilt afterwards from logs),
79//! `imported` (converted from another format, such as an Inspect `.eval`
80//! file or an EEE datastore entry). A reader comparing Cards can weigh a
81//! reconstructed Eval differently; the hub does not.
82//!
83//! # Conditions are per run; the header's facets are defaults
84//!
85//! One session may change the model, the temperature or the harness
86//! between runs, so the conditions a run was measured under belong to the
87//! run: [`crate::run::Run`] has the same six facets as the header. The
88//! header's facets are *defaults*. A run that omits a facet is given the
89//! header's value at the time it is written, and the copy is stored with
90//! the run; changing the header's defaults later does not change the
91//! conditions of runs already written. A facet the run carries itself wins
92//! over the default. Fingerprints are computed per run from the run's own
93//! facets; the header's fingerprint remains, as the fingerprint of the
94//! defaults.
95//!
96//! # Tasks and runs are not separate types
97//!
98//! There is no sample or task type beside the run. A run's `calls` log
99//! carries the inputs (the prompt, the reference) along with the outputs,
100//! which is enough material. A set of tasks with nothing executed yet is an
101//! Eval whose runs have empty outputs, and the link between a task Eval and
102//! an execution Eval is an ordinary relation between Evals
103//! (`core/derived_from`, `core/subset_of`). Repeated attempts at the same
104//! task (`trial.k`, a rerun after an error) are separate runs; overwriting a
105//! run is for correcting that same attempt. A producer that wants to prove
106//! two runs saw the same input puts a hash of it in the run's `ext`.
107//!
108//! # A run has no verdict
109//!
110//! A run records what happened ([`crate::run::RunStatus`]: `ok`, `error`,
111//! `skipped`) and what was measured directly ([`crate::run::Run::metrics`]).
112//! It never records whether it passed. Pass and fail depend on a grader and
113//! a threshold, and the same material can be graded several ways, so the
114//! verdict lives on the Card that graded it, in
115//! [`crate::card::Card::run_results`].
116//!
117//! # No grading facet
118//!
119//! An Eval (header and runs alike) has six of the seven facets. Grading
120//! belongs to the Card, because the same material can be graded several
121//! ways, and each grading is a different Card pointing at the same Eval.
122//!
123//! # Privacy across the relation
124//!
125//! A public Card may point at a private Eval. A reader without access sees
126//! that an Eval exists, its `version_id`, and the `content_hash` commitment —
127//! enough to know the Card is anchored to something specific — and nothing
128//! else. No title, no runs, no attachment URLs.
129//!
130//! The Card's own body does not reveal more than its edges. On every read
131//! that returns a body, a `relations[]` element whose target the reader may
132//! not see is removed and listed under `withheld.relations` as the same
133//! commitment (`type`, `version_id`, `content_hash`). The body returned is
134//! then not the stored one, and its `content_hash` does not match it. A
135//! query's `relations.to` condition matches only targets the caller may
136//! see, so a search cannot confirm a private name either.
137//!
138//! A write learns no more than a read. A target the writer may not see is
139//! stored unresolved, exactly as one that does not exist: the response and
140//! the `refs_resolved` badge are the same in both cases. The edge resolves
141//! for a reader once the target becomes visible to them.
142
143use schemars::JsonSchema;
144use serde::{Deserialize, Serialize};
145
146use crate::common::{Attachment, Ext, Producer, Redaction, Relation};
147use crate::facet::{Env, Generation, Harness, Model, Task, Trial};
148
149pub mod v1;
150
151/// An Eval header: what the material is, how it came to exist, and the
152/// default conditions for its runs. The runs themselves are not part of the
153/// body; they belong to the record and are written one by one
154/// (`schemas/run.json`).
155#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
156#[serde(deny_unknown_fields)]
157pub struct Eval {
158    /// Schema identifier; must be `evalhub.eval/2.0`.
159    #[schemars(extend("const" = "evalhub.eval/2.0"))]
160    pub schema: String,
161    /// Human-readable title.
162    pub title: String,
163    /// The software that wrote this record.
164    pub producer: Producer,
165    /// What the material is.
166    pub eval_kind: EvalKind,
167    /// How the material came to exist.
168    pub origin: Origin,
169    /// The model facet: the default for runs that do not carry their own.
170    #[serde(default, skip_serializing_if = "Option::is_none")]
171    pub model: Option<Model>,
172    /// The task facet: the default for runs that do not carry their own.
173    #[serde(default, skip_serializing_if = "Option::is_none")]
174    pub task: Option<Task>,
175    /// The harness facet: the default for runs that do not carry their own.
176    #[serde(default, skip_serializing_if = "Option::is_none")]
177    pub harness: Option<Harness>,
178    /// The generation facet: the default for runs that do not carry their own.
179    #[serde(default, skip_serializing_if = "Option::is_none")]
180    pub generation: Option<Generation>,
181    /// The trial facet: the default for runs that do not carry their own.
182    #[serde(default, skip_serializing_if = "Option::is_none")]
183    pub trial: Option<Trial>,
184    /// The environment facet: the default for runs that do not carry their own.
185    #[serde(default, skip_serializing_if = "Option::is_none")]
186    pub env: Option<Env>,
187    /// Edges to other Evals (`core/derived_from`, `core/subset_of`).
188    #[serde(default, skip_serializing_if = "Vec::is_empty")]
189    pub relations: Vec<Relation>,
190    /// Files this header refers to. A run's own files are in the run's
191    /// `attachments[]`, not here.
192    #[serde(default, skip_serializing_if = "Vec::is_empty")]
193    pub attachments: Vec<Attachment>,
194    /// What the producer removed before publishing.
195    #[serde(default, skip_serializing_if = "Option::is_none")]
196    pub redaction: Option<Redaction>,
197    /// Producer-private extensions keyed by namespace.
198    #[serde(default)]
199    pub ext: Ext,
200}
201
202/// What an Eval's material is.
203#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
204#[serde(rename_all = "snake_case")]
205pub enum EvalKind {
206    /// A set of runs, each with its model calls.
207    RunSet,
208    /// A set of prompts.
209    PromptSet,
210    /// A set of tasks.
211    TaskSet,
212    /// A set of traces.
213    TraceSet,
214}
215
216/// How an Eval's material came to exist.
217#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
218#[serde(rename_all = "snake_case")]
219pub enum Origin {
220    /// Captured while the evaluation ran.
221    Live,
222    /// Rebuilt afterwards from logs.
223    Reconstructed,
224    /// Converted from another format.
225    Imported,
226}