Skip to main content

evalhub_core/
lib.rs

1//! The domain rules of evalhub: what a record or a run must satisfy, what
2//! the hub derives from it, and nothing that touches a database or a
3//! socket.
4//!
5//! This crate is pure. Every function takes a value and returns a value. It
6//! knows the record types from [`evalhub_schema`] and it knows the rules in
7//! the design; it does not know that Postgres exists. That boundary is what
8//! makes the rules testable from fixtures alone (steps 1–3 of the build plan
9//! run with no database) and what keeps the store from growing opinions
10//! about records.
11//!
12//! # The model: records, and the runs of an Eval
13//!
14//! There are two record kinds, a Card and an Eval, each an append-only
15//! sequence of immutable versions under a name (the one exception is the
16//! 0.2.0 data migration of stored 1.0 bodies, which runs through
17//! [`eval::split_v1`] here and is `evalhub_store`'s to perform and
18//! audit). An Eval's versioned body is
19//! a *header*; its runs are rows of the Eval record, outside every version.
20//! Writing a run does not append a header version, and a header version
21//! does not rewrite a run.
22//!
23//! ```text
24//! Eval record {ns}/{name}
25//! ├── header versions @1, @2, …   validate::validate, content_hash, fingerprints
26//! └── runs, keyed by run_id        validate::run, run::materialise,
27//!                                  run::run_content_hash, fingerprint::for_run
28//!     runs_hash = run::runs_hash over every run of the record
29//! ```
30//!
31//! A run is not a record kind: it has no name, no versions and no
32//! visibility of its own, and it is addressed under its Eval. So it has its
33//! own functions here rather than a third [`evalhub_schema::RecordKind`].
34//! Its conditions are its own: the header's facets are defaults, copied
35//! onto a run that omits them when it is written ([`run::materialise`]),
36//! and a run is fingerprinted from its own facets ([`fingerprint::for_run`]).
37//! A run records what happened, never whether it passed; the verdict is a
38//! Card's `run_results`, whose form this crate checks and whose references
39//! to stored runs the store checks. The runs a Card judged, per Eval, are
40//! its *used set*: the `attrs.runs` of its `core/uses_eval` relations, or
41//! every run neither archived nor deleted when it is posted; this crate
42//! hashes a used set ([`run::runs_hash`]) but never computes one, because
43//! that needs the stored runs.
44//!
45//! # What the hub derives from a record
46//!
47//! On `POST`, the server calls into this crate in this order:
48//!
49//! ```text
50//! bytes ──▶ validate::structural   JSON Schema of the declared identifier, 2^53 check
51//!       ──▶ validate::semantic     counts, attachment paths, metric id form, run_results form
52//!       ──▶ canonical::canonicalize   RFC 8785 bytes  ──▶ content_hash (sha256)
53//!       ──▶ fingerprint::of_record    seven sha256s, one per facet (six for an Eval)
54//!       ──▶ badge::compute            refs_resolved? harness_registered? env_pinned? redacted?
55//!       ──▶ (store) one transaction
56//! ```
57//!
58//! All validation errors are collected, not short-circuited, and returned as
59//! one `422 { errors[] }` (see `evalhub_schema::error`). The record that
60//! reaches the store is the client's bytes, canonicalised; the hub adds
61//! facts around it and never rewrites it.
62//!
63//! One exception to "all errors": an `evalhub.eval/2.0` header that carries
64//! `runs` gets exactly one error, `runs_moved`, naming the run endpoints
65//! (`PUT /evals/{ns}/{name}/runs/{run_id}`,
66//! `POST /evals/{ns}/{name}/runs:batch`).
67//!
68//! # What the hub derives from a run
69//!
70//! On `PUT /evals/{ns}/{name}/runs/{run_id}` (and per element of a batch):
71//!
72//! ```text
73//! body ──▶ validate::run            run.json, 2^53, run_id, status ⇔ detail,
74//!                                   pointers into the run's own attachments, metrics keys
75//!      ──▶ run::materialise         copy the facets the run omits from the latest header
76//!      ──▶ run::run_content_hash    sha256(JCS({ …body, "run_id": … }))
77//!      ──▶ fingerprint::for_run     six sha256s
78//!      ──▶ (store) the row, then run::runs_hash over the record's runs
79//! ```
80//!
81//! # The 1.0 arm
82//!
83//! Release 0.2.0 still accepts an `evalhub.eval/1.0` body, with `runs[]` in
84//! it, for one release. [`eval::declared_schema`] tells the caller which
85//! arm a body declares; [`validate()`] checks a 1.0 body as 0.1.x did,
86//! including that its runs' `calls` / `artifacts[]` name the body's
87//! attachments; and [`eval::split_v1`] converts it into the 2.0 header and
88//! runs a 2.0 producer would have written. The ingest of a 1.0 body and the
89//! data migration of stored 1.0 versions both use that one function.
90//! Release 0.3.0 removes the arm.
91//!
92//! # Hashes
93//!
94//! Every hash is SHA-256 over RFC 8785 (JCS) canonical JSON, hex-encoded in
95//! lower case. They are formulas a client can recompute, not opaque ids:
96//!
97//! | Hash                  | Formula                                                                   |
98//! | --------------------- | ------------------------------------------------------------------------- |
99//! | record `content_hash` | `sha256(JCS(body))` ([`canonical::hash_value`])                            |
100//! | run `content_hash`    | `sha256(JCS({ …body, "run_id": run_id }))`, body after materialisation ([`run::run_content_hash`]) |
101//! | `runs_hash`           | `sha256(JCS([[run_id, hex(content_hash)], …]))`, sorted by `run_id` ascending, every run of the record including archived and deleted ones ([`run::runs_hash`]) |
102//! | used-set hash         | the same function as `runs_hash`, over the runs of one Eval a Card used    |
103//!
104//! Archiving or deleting a run changes none of them, as tombstoning a
105//! version does not change its `content_hash`: the material is the same,
106//! only how it is shown and kept has changed. A header's hash changes when
107//! a header version is appended; a run's when its content is overwritten;
108//! `runs_hash` when a run is added or overwritten; a used-set hash when a
109//! run in the used set is overwritten.
110//!
111//! # Invariants this crate is responsible for
112//!
113//! - **Canonical form is total and deterministic.** The same JSON value, in
114//!   any key order and any whitespace, canonicalises to the same bytes, and
115//!   therefore the same `content_hash`. This is what makes `POST` idempotent:
116//!   the store compares the new hash with the latest version's and returns
117//!   `200` with the existing version if they match. The same holds for a
118//!   run's hash and an identical `PUT`.
119//! - **A fingerprint depends only on the facet's core keys.** `ext` and keys
120//!   marked `x-fingerprint: false` in the schema are removed before hashing.
121//!   The exclusion is read from the schema, so the schema is the single
122//!   source for what "the same conditions" means.
123//! - **Validation is the same everywhere.** The server, a future CLI
124//!   `evalhub check`, and a harness embedding this crate all reject the same
125//!   records and runs for the same codes.
126//! - **One conversion from 1.0.** A 1.0 run converted at ingest and one
127//!   converted by the migration are the same bytes and the same hash,
128//!   because both go through [`eval::split_v1`].
129//!
130//! # What this crate refuses to do
131//!
132//! - It does not open attachments. `samples_ref` must name an entry in
133//!   `attachments[]`; whether the file has the rows it claims is not checked
134//!   (design decision: content validation is out of scope for v0 and, when
135//!   it comes, it is a separate job, not part of ingest).
136//! - It does not decide whether two records are "the same evaluation". It
137//!   produces per-facet fingerprints; the reader compares them.
138//! - It does not resolve relations. Resolution needs the store; this crate
139//!   only checks the *form* of a reference and reports, via the badge input,
140//!   whether the store said it resolved.
141//! - It does not know which runs exist. Whether a `run_results[].run_id`
142//!   names a stored run, and whether it is in the Card's used set, is the
143//!   store's check (`run_unknown`, `run_not_in_used_set`); how many
144//!   `run_results` a Card may carry is the server's (`too_many_run_results`),
145//!   as are the request body and batch limits.
146//! - It does not know who is asking. Visibility (runs follow their Eval,
147//!   `run_results` follow their Card, and entries naming an Eval the
148//!   reader may not see are withheld) is decided by the store and the
149//!   server; every function here answers the same for every caller.
150//!
151//! # Modules
152//!
153//! - [`canonical`] — RFC 8785 serialisation and the content hash.
154//! - [`fingerprint`] — per-facet hashing with schema-driven exclusion, for
155//!   records and runs.
156//! - [`mod@validate`] — structural and semantic checks, error collection,
157//!   for records and runs.
158//! - [`eval`] — the two Eval schema arms and the 1.0 → 2.0 conversion.
159//! - [`run`] — facet materialisation, the run content hash and `runs_hash`.
160//! - [`badge`] — the badge rules and their inputs.
161//! - [`registry`] — the `core/` registry entries (metrics, relation types)
162//!   that ship with the hub and are read-only.
163//! - [`id`] — ULID generation and the version identifier.
164
165pub mod badge;
166pub mod canonical;
167pub mod eval;
168pub mod fingerprint;
169pub mod id;
170pub mod registry;
171pub mod run;
172pub mod validate;
173
174pub use badge::{Badge, BadgeInput, compute as compute_badges};
175pub use canonical::{ContentHash, canonicalize, content_hash};
176pub use eval::{EvalSchema, SplitV1, declared_schema, split_v1};
177pub use fingerprint::{Facet, Fingerprints, fingerprints};
178pub use id::{RecordId, VersionId};
179pub use run::{materialise, run_content_hash, runs_hash};
180pub use validate::validate;