Expand description
Validation: everything the hub checks before it stores a record or a run.
Validation runs in two passes and collects every failure from both into one list, so a producer sees all problems in one round trip. There are two entry points:
validatefor a record (a Card, or an Eval header), keyed byRecordKind;run()for one run of an Eval, given therun_idit is written under.
A run is not a RecordKind. RecordKind is the two kinds that appear
in URLs and in the store’s type column, and every match on it is
exhaustive; a run is a row of an Eval record, addressed under it, so it
has its own entry point rather than a third variant that every one of
those matches would have to reject.
§Pass 1: structural
The value is checked against a JSON Schema using jsonschema with
draft 2020-12. The core is closed, so an unknown key anywhere outside
ext is a schema error. In the same pass every number is inspected:
an integer outside ±2^53 is number_too_large.
Which schema: a record is checked against the document for the
identifier its own schema key declares, through
RecordKind::schema_for:
Declared schema | Document |
|---|---|
evalhub.card/1.1, evalhub.card/1.0 | schemas/card.json |
evalhub.eval/2.0 | schemas/eval-2.json |
evalhub.eval/1.0 | schemas/eval.json |
anything else, or no schema key | the kind’s current one |
The last row is what makes an unknown identifier an error: the current
document’s schema key is a constant, so the value fails it at
/schema (schema), exactly as it did before a kind had two
identifiers. A run is checked against schemas/run.json.
The schemas are the generated ones from evalhub_schema (the same
generator output that is committed under schemas/). Each is compiled
once per process, one validator per accepted identifier plus one for
the run, on first use. The validators are built in offline mode: every
$ref in the schema is local (#/$defs/…), and a reference that would
need fetching is a build error, not a fetch.
§The two Eval arms
Release 0.2.0 accepts two majors of the Eval schema
(crate::eval::EvalSchema):
- 2.0 (
evalhub.eval/2.0) is a header. Runs are not part of it; they are written one by one (PUT /evals/{ns}/{name}/runs/{run_id}) or in a batch (POST /evals/{ns}/{name}/runs:batch). A 2.0 body that has arunskey at all, whatever its value, is answered with one error,runs_movedat/runs, before the structural pass, and nothing else: every other error the body might have would be noise next to “this is the wrong endpoint for your runs”, and the producer that sendsrunsis typically a 1.0 producer that bumped the identifier. - 1.0 (
evalhub.eval/1.0) is the old shape withruns[]in the body, validated as 0.1.x validated it, including the check that eachruns[].callsandruns[].artifacts[]names anattachments[].pathof the body. That check exists for the 1.0 arm only. A 1.0 body that passes is converted into a 2.0 header and its runs bycrate::eval::split_v1; release 0.3.0 removes the arm.
§Pass 2: semantic
Rules the schema language cannot express. On a record:
| Rule | Code |
|---|---|
results non-empty ⇒ counts present | counts_missing |
counts.attempted >= completed + failed + skipped + errored | counts_inconsistent |
every results[].samples_ref is an attachments[].path | attachment_ref_unknown |
Eval 1.0 only: every runs[].calls, runs[].artifacts[] is an attachments[].path | attachment_ref_unknown |
attachments[].path unique, relative, contains no .. | attachment_path_invalid |
results[].metric matches {ns}/{name} | metric_id_invalid |
run_results[].metric matches {ns}/{name} | metric_id_invalid |
run_results[] element has value or label (or both) | run_result_value_or_label |
run_results[].eval is the {ns}/{name} of a core/uses_eval relation’s target | run_results_eval_unknown |
relations[].type matches {ns}/{name} | schema |
relations[].to is {ns}/{name}@{seq}, external:…, or hf:… | schema |
every key of ext, and of each facet’s ext, matches {ns}/{name} | ext_key_invalid |
run_results[].eval is compared with the target of each of the Card’s
core/uses_eval relations without its @{seq}: runs belong to the
Eval record, not to a version, so a verdict names the record. Only a
hub target ({ns}/{name}@{seq}) can match; an external: or hf:
target has no runs on this hub. Whether that target resolves, and
whether the writer may see it, is not this crate’s question.
Three more rules about run_results need data this crate does not
have, and are checked elsewhere:
| Rule | Code | Checked by |
|---|---|---|
every run_id of a used set has a row in its Eval (archived and deleted rows count) | run_unknown | the store |
every run_results[].run_id is in the used set of its eval | run_not_in_used_set | the store |
run_results[] is no longer than the configured limit | too_many_run_results | the server |
On a run (run()), everything is checked against the run itself; a
run never points into the header:
| Rule | Code |
|---|---|
the given run_id is non-empty, has no /, is at most 200 bytes | run_id_invalid |
the body’s run_id equals the given one | run_id_mismatch |
error present iff status is error; skip_reason iff status is skipped | run_status_detail |
calls, artifacts[], error.log each name an attachments[].path of the run | attachment_ref_unknown |
the run’s attachments[].path unique, relative, contains no .. | attachment_path_invalid |
every key of metrics matches {ns}/{name} | metric_id_invalid |
every key of ext, and of each facet’s ext, matches {ns}/{name} | ext_key_invalid |
ext is treated on a run as on a record: the map itself is open, and
only its keys are checked, at the top level and inside each facet that
carries one (model, task, harness, generation, trial,
grading, env), one error per key at /ext/{key} or
/{facet}/ext/{key}. The values are never looked into, and neither is
meta.
The semantic pass runs on whatever is present even when the structural
pass failed, so a producer sees both kinds of problem at once. Errors
are sorted by (path, code); the order is deterministic.
§What is deliberately not an error
- A
relations[].tothat does not resolve. The record is accepted; therefs_resolvedbadge is withheld. A Card may legitimately be published before the Eval it cites. - A
harnessormetricthe registry does not know, including a run’smetricskey. Accepted; theharness_registered/metric_registeredbadges are withheld. The hub does not gate on vocabulary. - An
attachments[].sha256that has not been uploaded. This is not a validation failure but a state conflict,409 attachment_missing, raised by the store, because only the store knows. - An Eval header with zero runs, of any
eval_kind. A header is complete on its own; its runs are written afterwards.
§Output
Vec<ErrorEntry { path, code, hint }>, empty on success. path is a
JSON pointer into the submitted value (/results/0/metric; the root is
""). hint is prose and not part of the contract.
Enums§
- Relation
Target - Where a relation points.
Constants§
- MAX_
RUN_ ID_ BYTES - Longest
run_idaccepted, in bytes of UTF-8.
Functions§
- is_
valid_ id - Whether
shas the form{ns}/{name}: exactly one/, both parts non-empty, each starting with[a-z0-9]and continuing with[a-z0-9._-]. - is_
valid_ run_ id - Whether
sis a validrun_id: non-empty, no/, at mostMAX_RUN_ID_BYTESbytes of UTF-8. Nothing else is imposed (seeevalhub_schema::run); everyrun_ida 0.1.x producer wrote passes. - parse_
relation_ target - Parse a
relations[].tovalue.Nonewhen it has none of the accepted forms. - run
- Validate one run of an Eval, written under
run_id(the path segment ofPUT /evals/{ns}/{name}/runs/{run_id}, or the element’s ownrun_idin a batch). Returns every failure, sorted by(path, code); empty when the run is accepted. - validate
- Validate a record of
kindand return every failure, sorted by(path, code); empty when the record is accepted.