1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
//! The domain rules of evalhub: what a record or a run must satisfy, what
//! the hub derives from it, and nothing that touches a database or a
//! socket.
//!
//! This crate is pure. Every function takes a value and returns a value. It
//! knows the record types from [`evalhub_schema`] and it knows the rules in
//! the design; it does not know that Postgres exists. That boundary is what
//! makes the rules testable from fixtures alone (steps 1–3 of the build plan
//! run with no database) and what keeps the store from growing opinions
//! about records.
//!
//! # The model: records, and the runs of an Eval
//!
//! There are two record kinds, a Card and an Eval, each an append-only
//! sequence of immutable versions under a name (the one exception is the
//! 0.2.0 data migration of stored 1.0 bodies, which is `evalhub_store`'s
//! to perform and audit). An Eval's versioned body is a *header*; its runs
//! are rows of the Eval record, outside every version. Writing a run does
//! not append a header version, and a header version does not rewrite a
//! run.
//!
//! ```text
//! Eval record {ns}/{name}
//! ├── header versions @1, @2, … validate::validate, content_hash, fingerprints
//! └── runs, keyed by run_id validate::run, run::materialise,
//! run::run_content_hash, fingerprint::for_run
//! runs_hash = run::runs_hash over every run of the record
//! ```
//!
//! A run is not a record kind: it has no name, no versions and no
//! visibility of its own, and it is addressed under its Eval. So it has its
//! own functions here rather than a third [`evalhub_schema::RecordKind`].
//! Its conditions are its own: the header's facets are defaults, copied
//! onto a run that omits them when it is written ([`run::materialise`]),
//! and a run is fingerprinted from its own facets ([`fingerprint::for_run`]).
//! A run records what happened, never whether it passed; the verdict is a
//! Card's `run_results`, whose form this crate checks and whose references
//! to stored runs the store checks. The runs a Card judged, per Eval, are
//! its *used set*: the `attrs.runs` of its `core/uses_eval` relations, or
//! every run neither archived nor deleted when it is posted; this crate
//! hashes a used set ([`run::runs_hash`]) but never computes one, because
//! that needs the stored runs.
//!
//! # What the hub derives from a record
//!
//! On `POST`, the server calls into this crate in this order:
//!
//! ```text
//! bytes ──▶ validate::structural JSON Schema of the declared identifier, 2^53 check
//! ──▶ validate::semantic counts, attachment paths, metric id form, run_results form
//! ──▶ canonical::canonicalize RFC 8785 bytes ──▶ content_hash (sha256)
//! ──▶ fingerprint::of_record seven sha256s, one per facet (six for an Eval)
//! ──▶ badge::compute refs_resolved? harness_registered? env_pinned? redacted?
//! ──▶ (store) one transaction
//! ```
//!
//! All validation errors are collected, not short-circuited, and returned as
//! one `422 { errors[] }` (see `evalhub_schema::error`). The record that
//! reaches the store is the client's bytes, canonicalised; the hub adds
//! facts around it and never rewrites it.
//!
//! One exception to "all errors": an `evalhub.eval/2.0` header that carries
//! `runs`, and any body that declares `evalhub.eval/1.0`, gets exactly one
//! error, `runs_moved`, naming the run endpoints
//! (`PUT /evals/{ns}/{name}/runs/{run_id}`,
//! `POST /evals/{ns}/{name}/runs:batch`).
//!
//! # What the hub derives from a run
//!
//! On `PUT /evals/{ns}/{name}/runs/{run_id}` (and per element of a batch):
//!
//! ```text
//! body ──▶ validate::run run.json, 2^53, run_id, status ⇔ detail,
//! pointers into the run's own attachments, metrics keys
//! ──▶ run::materialise copy the facets the run omits from the latest header
//! ──▶ run::run_content_hash sha256(JCS({ …body, "run_id": … }))
//! ──▶ fingerprint::for_run six sha256s
//! ──▶ (store) the row, then run::runs_hash over the record's runs
//! ```
//!
//! # `evalhub.eval/1.0`
//!
//! Releases 0.2.0 and 0.3.0 accepted an `evalhub.eval/1.0` body, with
//! `runs[]` in it, and converted it at ingest. Release 0.4.0 closed that
//! window: [`validate()`] answers a body declaring `evalhub.eval/1.0` with
//! the single `runs_moved` error, and this crate no longer knows the 1.0
//! shape or its conversion. The conversion of stored 0.1.x bodies belongs
//! to `evalhub_store`'s data migration, which keeps its own copy.
//!
//! # Hashes
//!
//! Every hash is SHA-256 over RFC 8785 (JCS) canonical JSON, hex-encoded in
//! lower case. They are formulas a client can recompute, not opaque ids:
//!
//! | Hash | Formula |
//! | --------------------- | ------------------------------------------------------------------------- |
//! | record `content_hash` | `sha256(JCS(body))` ([`canonical::hash_value`]) |
//! | run `content_hash` | `sha256(JCS({ …body, "run_id": run_id }))`, body after materialisation ([`run::run_content_hash`]) |
//! | `runs_hash` | `sha256(JCS([[run_id, hex(content_hash)], …]))`, sorted by `run_id` ascending, every run of the record including archived and deleted ones ([`run::runs_hash`]) |
//! | used-set hash | the same function as `runs_hash`, over the runs of one Eval a Card used |
//!
//! Archiving or deleting a run changes none of them, as tombstoning a
//! version does not change its `content_hash`: the material is the same,
//! only how it is shown and kept has changed. A header's hash changes when
//! a header version is appended; a run's when its content is overwritten;
//! `runs_hash` when a run is added or overwritten; a used-set hash when a
//! run in the used set is overwritten.
//!
//! # Invariants this crate is responsible for
//!
//! - **Canonical form is total and deterministic.** The same JSON value, in
//! any key order and any whitespace, canonicalises to the same bytes, and
//! therefore the same `content_hash`. This is what makes `POST` idempotent:
//! the store compares the new hash with the latest version's and returns
//! `200` with the existing version if they match. The same holds for a
//! run's hash and an identical `PUT`.
//! - **A fingerprint depends only on the facet's core keys.** `ext` and keys
//! marked `x-fingerprint: false` in the schema are removed before hashing.
//! The exclusion is read from the schema, so the schema is the single
//! source for what "the same conditions" means.
//! - **Validation is the same everywhere.** The server, a future CLI
//! `evalhub check`, and a harness embedding this crate all reject the same
//! records and runs for the same codes.
//!
//! # What this crate refuses to do
//!
//! - It does not open attachments. `samples_ref` must name an entry in
//! `attachments[]`; whether the file has the rows it claims is not checked
//! (design decision: content validation is out of scope for v0 and, when
//! it comes, it is a separate job, not part of ingest).
//! - It does not decide whether two records are "the same evaluation". It
//! produces per-facet fingerprints; the reader compares them.
//! - It does not resolve relations. Resolution needs the store; this crate
//! only checks the *form* of a reference and reports, via the badge input,
//! whether the store said it resolved.
//! - It does not know which runs exist. Whether a `run_results[].run_id`
//! names a stored run, and whether it is in the Card's used set, is the
//! store's check (`run_unknown`, `run_not_in_used_set`); how many
//! `run_results` a Card may carry is the server's (`too_many_run_results`),
//! as are the request body and batch limits.
//! - It does not know who is asking. Visibility (runs follow their Eval,
//! `run_results` follow their Card, and entries naming an Eval the
//! reader may not see are withheld) is decided by the store and the
//! server; every function here answers the same for every caller.
//!
//! # Modules
//!
//! - [`canonical`] — RFC 8785 serialisation and the content hash.
//! - [`fingerprint`] — per-facet hashing with schema-driven exclusion, for
//! records and runs.
//! - [`mod@validate`] — structural and semantic checks, error collection,
//! for records and runs.
//! - [`run`] — facet materialisation, the run content hash and `runs_hash`.
//! - [`badge`] — the badge rules and their inputs.
//! - [`registry`] — the `core/` registry entries (metrics, relation types)
//! that ship with the hub and are read-only.
//! - [`id`] — ULID generation and the version identifier.
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use validate;