Skip to main content

Module eval

Module eval 

Source
Expand description

The two Eval schema arms, and the one 1.0 → 2.0 conversion.

Release 0.2.0 accepts two majors of the Eval record schema (see “Versioning of the schema itself” in evalhub_schema):

evalhub.eval/1.0   header + runs[] in one body          (accepted by 0.2.0 only)
evalhub.eval/2.0   header only; runs are rows of the record, written apart

declared_schema says which arm a body declares, so that the validator, the store and the server agree on it without each parsing schema for themselves. split_v1 turns a 1.0 body into what a 2.0 producer would have sent: a header and one run per runs[] element. It is the only implementation of that conversion; the ingest of a 1.0 body and the one-shot data migration of stored 1.0 versions both call it, so a run converted at ingest and a run converted by the migration are the same run, byte for byte, and have the same content hash.

§The conversion

1.0 body ──▶ header: body − runs, schema := "evalhub.eval/2.0"
         └─▶ runs[i] ──▶ run: run_id, status (+ error / skip_reason),
                              started_at, ended_at, calls, artifacts   (verbatim)
                              attachments   the header entries calls/artifacts name
                              meta          { "v0.2.0_migration": { "outcome": … } }
                              facets        materialised from the header

Nothing else in the header changes: its attachments[] keep the entries the runs point at (they are copied, not moved), so the header is exactly the 1.0 body minus runs under the 2.0 identifier, and its content hash is the hash of that header posted as 2.0.

outcome mixed the execution state with a verdict; 2.0 keeps only the state:

1.0 outcome2.0 statusdetail
passok
failok
errorerrorerror: { "kind": "unrecorded" }
skippedskippedskip_reason: "unrecorded"

A 1.0 run says that it errored or was skipped but never why, so the conversion writes unrecorded rather than a guess. The dropped outcome is kept verbatim in the run’s meta, under V1_MIGRATION_META_KEY (v0.2.0_migration), so the verdict a 0.1.x producer wrote is not lost; it is simply no longer the hub’s to interpret. A later release that has to drop a meaning again keeps it the same way, under its own v{version}_migration key.

The run’s facets are the header’s, copied onto the run (crate::run::materialise); split_v1 owns materialisation on this path, so the run write path’s own materialise, which is idempotent, finds nothing left to copy.

A 1.0 run’s calls and artifacts[] named paths in the body’s attachments[]. A 2.0 run points only into its own, so each header entry a run names is copied into that run’s attachments[], in the header’s order, once.

Structs§

SplitV1
A 1.0 body split into its 2.0 parts. See split_v1.

Enums§

EvalSchema
Which major of the Eval schema a body declares.

Constants§

UNRECORDED
The value the conversion writes where 1.0 recorded no reason: the error.kind of an error run, the skip_reason of a skipped one.
V1_MIGRATION_META_KEY
The meta key under which the 1.0 → 2.0 conversion keeps what it dropped: { "v0.2.0_migration": { "outcome": … } }.

Functions§

declared_schema
The Eval arm body declares in its schema key, or None when the key is absent, not a string, or not an accepted Eval identifier.
split_v1
Split an evalhub.eval/1.0 body into a 2.0 header and its runs.