evalhub_core/lib.rs
1//! The domain rules of evalhub: what a record or a run must satisfy, what
2//! the hub derives from it, and nothing that touches a database or a
3//! socket.
4//!
5//! This crate is pure. Every function takes a value and returns a value. It
6//! knows the record types from [`evalhub_schema`] and it knows the rules in
7//! the design; it does not know that Postgres exists. That boundary is what
8//! makes the rules testable from fixtures alone (steps 1–3 of the build plan
9//! run with no database) and what keeps the store from growing opinions
10//! about records.
11//!
12//! # The model: records, and the runs of an Eval
13//!
14//! There are two record kinds, a Card and an Eval, each an append-only
15//! sequence of immutable versions under a name (the one exception is the
16//! 0.2.0 data migration of stored 1.0 bodies, which is `evalhub_store`'s
17//! to perform and audit). An Eval's versioned body is a *header*; its runs
18//! are rows of the Eval record, outside every version. Writing a run does
19//! not append a header version, and a header version does not rewrite a
20//! run.
21//!
22//! ```text
23//! Eval record {ns}/{name}
24//! ├── header versions @1, @2, … validate::validate, content_hash, fingerprints
25//! └── runs, keyed by run_id validate::run, run::materialise,
26//! run::run_content_hash, fingerprint::for_run
27//! runs_hash = run::runs_hash over every run of the record
28//! ```
29//!
30//! A run is not a record kind: it has no name, no versions and no
31//! visibility of its own, and it is addressed under its Eval. So it has its
32//! own functions here rather than a third [`evalhub_schema::RecordKind`].
33//! Its conditions are its own: the header's facets are defaults, copied
34//! onto a run that omits them when it is written ([`run::materialise`]),
35//! and a run is fingerprinted from its own facets ([`fingerprint::for_run`]).
36//! A run records what happened, never whether it passed; the verdict is a
37//! Card's `run_results`, whose form this crate checks and whose references
38//! to stored runs the store checks. The runs a Card judged, per Eval, are
39//! its *used set*: the `attrs.runs` of its `core/uses_eval` relations, or
40//! every run neither archived nor deleted when it is posted; this crate
41//! hashes a used set ([`run::runs_hash`]) but never computes one, because
42//! that needs the stored runs.
43//!
44//! # What the hub derives from a record
45//!
46//! On `POST`, the server calls into this crate in this order:
47//!
48//! ```text
49//! bytes ──▶ validate::structural JSON Schema of the declared identifier, 2^53 check
50//! ──▶ validate::semantic counts, attachment paths, metric id form, run_results form
51//! ──▶ canonical::canonicalize RFC 8785 bytes ──▶ content_hash (sha256)
52//! ──▶ fingerprint::of_record seven sha256s, one per facet (six for an Eval)
53//! ──▶ badge::compute refs_resolved? harness_registered? env_pinned? redacted?
54//! ──▶ (store) one transaction
55//! ```
56//!
57//! All validation errors are collected, not short-circuited, and returned as
58//! one `422 { errors[] }` (see `evalhub_schema::error`). The record that
59//! reaches the store is the client's bytes, canonicalised; the hub adds
60//! facts around it and never rewrites it.
61//!
62//! One exception to "all errors": an `evalhub.eval/2.0` header that carries
63//! `runs`, and any body that declares `evalhub.eval/1.0`, gets exactly one
64//! error, `runs_moved`, naming the run endpoints
65//! (`PUT /evals/{ns}/{name}/runs/{run_id}`,
66//! `POST /evals/{ns}/{name}/runs:batch`).
67//!
68//! # What the hub derives from a run
69//!
70//! On `PUT /evals/{ns}/{name}/runs/{run_id}` (and per element of a batch):
71//!
72//! ```text
73//! body ──▶ validate::run run.json, 2^53, run_id, status ⇔ detail,
74//! pointers into the run's own attachments, metrics keys
75//! ──▶ run::materialise copy the facets the run omits from the latest header
76//! ──▶ run::run_content_hash sha256(JCS({ …body, "run_id": … }))
77//! ──▶ fingerprint::for_run six sha256s
78//! ──▶ (store) the row, then run::runs_hash over the record's runs
79//! ```
80//!
81//! # `evalhub.eval/1.0`
82//!
83//! Releases 0.2.0 and 0.3.0 accepted an `evalhub.eval/1.0` body, with
84//! `runs[]` in it, and converted it at ingest. Release 0.4.0 closed that
85//! window: [`validate()`] answers a body declaring `evalhub.eval/1.0` with
86//! the single `runs_moved` error, and this crate no longer knows the 1.0
87//! shape or its conversion. The conversion of stored 0.1.x bodies belongs
88//! to `evalhub_store`'s data migration, which keeps its own copy.
89//!
90//! # Hashes
91//!
92//! Every hash is SHA-256 over RFC 8785 (JCS) canonical JSON, hex-encoded in
93//! lower case. They are formulas a client can recompute, not opaque ids:
94//!
95//! | Hash | Formula |
96//! | --------------------- | ------------------------------------------------------------------------- |
97//! | record `content_hash` | `sha256(JCS(body))` ([`canonical::hash_value`]) |
98//! | run `content_hash` | `sha256(JCS({ …body, "run_id": run_id }))`, body after materialisation ([`run::run_content_hash`]) |
99//! | `runs_hash` | `sha256(JCS([[run_id, hex(content_hash)], …]))`, sorted by `run_id` ascending, every run of the record including archived and deleted ones ([`run::runs_hash`]) |
100//! | used-set hash | the same function as `runs_hash`, over the runs of one Eval a Card used |
101//!
102//! Archiving or deleting a run changes none of them, as tombstoning a
103//! version does not change its `content_hash`: the material is the same,
104//! only how it is shown and kept has changed. A header's hash changes when
105//! a header version is appended; a run's when its content is overwritten;
106//! `runs_hash` when a run is added or overwritten; a used-set hash when a
107//! run in the used set is overwritten.
108//!
109//! # Invariants this crate is responsible for
110//!
111//! - **Canonical form is total and deterministic.** The same JSON value, in
112//! any key order and any whitespace, canonicalises to the same bytes, and
113//! therefore the same `content_hash`. This is what makes `POST` idempotent:
114//! the store compares the new hash with the latest version's and returns
115//! `200` with the existing version if they match. The same holds for a
116//! run's hash and an identical `PUT`.
117//! - **A fingerprint depends only on the facet's core keys.** `ext` and keys
118//! marked `x-fingerprint: false` in the schema are removed before hashing.
119//! The exclusion is read from the schema, so the schema is the single
120//! source for what "the same conditions" means.
121//! - **Validation is the same everywhere.** The server, a future CLI
122//! `evalhub check`, and a harness embedding this crate all reject the same
123//! records and runs for the same codes.
124//!
125//! # What this crate refuses to do
126//!
127//! - It does not open attachments. `samples_ref` must name an entry in
128//! `attachments[]`; whether the file has the rows it claims is not checked
129//! (design decision: content validation is out of scope for v0 and, when
130//! it comes, it is a separate job, not part of ingest).
131//! - It does not decide whether two records are "the same evaluation". It
132//! produces per-facet fingerprints; the reader compares them.
133//! - It does not resolve relations. Resolution needs the store; this crate
134//! only checks the *form* of a reference and reports, via the badge input,
135//! whether the store said it resolved.
136//! - It does not know which runs exist. Whether a `run_results[].run_id`
137//! names a stored run, and whether it is in the Card's used set, is the
138//! store's check (`run_unknown`, `run_not_in_used_set`); how many
139//! `run_results` a Card may carry is the server's (`too_many_run_results`),
140//! as are the request body and batch limits.
141//! - It does not know who is asking. Visibility (runs follow their Eval,
142//! `run_results` follow their Card, and entries naming an Eval the
143//! reader may not see are withheld) is decided by the store and the
144//! server; every function here answers the same for every caller.
145//!
146//! # Modules
147//!
148//! - [`canonical`] — RFC 8785 serialisation and the content hash.
149//! - [`fingerprint`] — per-facet hashing with schema-driven exclusion, for
150//! records and runs.
151//! - [`mod@validate`] — structural and semantic checks, error collection,
152//! for records and runs.
153//! - [`run`] — facet materialisation, the run content hash and `runs_hash`.
154//! - [`badge`] — the badge rules and their inputs.
155//! - [`registry`] — the `core/` registry entries (metrics, relation types)
156//! that ship with the hub and are read-only.
157//! - [`id`] — ULID generation and the version identifier.
158
159pub mod badge;
160pub mod canonical;
161pub mod fingerprint;
162pub mod id;
163pub mod registry;
164pub mod run;
165pub mod validate;
166
167pub use badge::{Badge, BadgeInput, compute as compute_badges};
168pub use canonical::{ContentHash, canonicalize, content_hash};
169pub use fingerprint::{Facet, Fingerprints, fingerprints};
170pub use id::{RecordId, VersionId};
171pub use run::{materialise, run_content_hash, runs_hash};
172pub use validate::validate;