Skip to main content

Module validate

Module validate 

Source
Expand description

Validation: everything the hub checks before it stores a record or a run.

Validation runs in two passes and collects every failure from both into one list, so a producer sees all problems in one round trip. There are two entry points:

  • validate for a record (a Card, or an Eval header), keyed by RecordKind;
  • run() for one run of an Eval, given the run_id it is written under.

A run is not a RecordKind. RecordKind is the two kinds that appear in URLs and in the store’s type column, and every match on it is exhaustive; a run is a row of an Eval record, addressed under it, so it has its own entry point rather than a third variant that every one of those matches would have to reject.

§Pass 1: structural

The value is checked against a JSON Schema using jsonschema with draft 2020-12. The core is closed, so an unknown key anywhere outside ext is a schema error. In the same pass every number is inspected: an integer outside ±2^53 is number_too_large.

Which schema: a record is checked against the document for the identifier its own schema key declares, through RecordKind::schema_for:

Declared schemaDocument
evalhub.card/1.1, evalhub.card/1.0schemas/card.json
evalhub.eval/2.0schemas/eval-2.json
evalhub.eval/1.0schemas/eval.json
anything else, or no schema keythe kind’s current one

The last row is what makes an unknown identifier an error: the current document’s schema key is a constant, so the value fails it at /schema (schema), exactly as it did before a kind had two identifiers. A run is checked against schemas/run.json.

The schemas are the generated ones from evalhub_schema (the same generator output that is committed under schemas/). Each is compiled once per process, one validator per accepted identifier plus one for the run, on first use. The validators are built in offline mode: every $ref in the schema is local (#/$defs/…), and a reference that would need fetching is a build error, not a fetch.

§The two Eval arms

Release 0.2.0 accepts two majors of the Eval schema (crate::eval::EvalSchema):

  • 2.0 (evalhub.eval/2.0) is a header. Runs are not part of it; they are written one by one (PUT /evals/{ns}/{name}/runs/{run_id}) or in a batch (POST /evals/{ns}/{name}/runs:batch). A 2.0 body that has a runs key at all, whatever its value, is answered with one error, runs_moved at /runs, before the structural pass, and nothing else: every other error the body might have would be noise next to “this is the wrong endpoint for your runs”, and the producer that sends runs is typically a 1.0 producer that bumped the identifier.
  • 1.0 (evalhub.eval/1.0) is the old shape with runs[] in the body, validated as 0.1.x validated it, including the check that each runs[].calls and runs[].artifacts[] names an attachments[].path of the body. That check exists for the 1.0 arm only. A 1.0 body that passes is converted into a 2.0 header and its runs by crate::eval::split_v1; release 0.3.0 removes the arm.

§Pass 2: semantic

Rules the schema language cannot express. On a record:

RuleCode
results non-empty ⇒ counts presentcounts_missing
counts.attempted >= completed + failed + skipped + erroredcounts_inconsistent
every results[].samples_ref is an attachments[].pathattachment_ref_unknown
Eval 1.0 only: every runs[].calls, runs[].artifacts[] is an attachments[].pathattachment_ref_unknown
attachments[].path unique, relative, contains no ..attachment_path_invalid
results[].metric matches {ns}/{name}metric_id_invalid
run_results[].metric matches {ns}/{name}metric_id_invalid
run_results[] element has value or label (or both)run_result_value_or_label
run_results[].eval is the {ns}/{name} of a core/uses_eval relation’s targetrun_results_eval_unknown
relations[].type matches {ns}/{name}schema
relations[].to is {ns}/{name}@{seq}, external:…, or hf:…schema

run_results[].eval is compared with the target of each of the Card’s core/uses_eval relations without its @{seq}: runs belong to the Eval record, not to a version, so a verdict names the record. Only a hub target ({ns}/{name}@{seq}) can match; an external: or hf: target has no runs on this hub. Whether that target resolves, and whether the writer may see it, is not this crate’s question.

Three more rules about run_results need data this crate does not have, and are checked elsewhere:

RuleCodeChecked by
every run_id of a used set has a row in its Eval (archived and deleted rows count)run_unknownthe store
every run_results[].run_id is in the used set of its evalrun_not_in_used_setthe store
run_results[] is no longer than the configured limittoo_many_run_resultsthe server

On a run (run()), everything is checked against the run itself; a run never points into the header:

RuleCode
the given run_id is non-empty, has no /, is at most 200 bytesrun_id_invalid
the body’s run_id equals the given onerun_id_mismatch
error present iff status is error; skip_reason iff status is skippedrun_status_detail
calls, artifacts[], error.log each name an attachments[].path of the runattachment_ref_unknown
the run’s attachments[].path unique, relative, contains no ..attachment_path_invalid
every key of metrics matches {ns}/{name}metric_id_invalid

ext is treated on a run as on a record: an object whose keys are meant to be namespaces, accepted as the JSON Schema describes it, with no semantic rule of its own in this crate today. meta is never looked into.

The semantic pass runs on whatever is present even when the structural pass failed, so a producer sees both kinds of problem at once. Errors are sorted by (path, code); the order is deterministic.

§What is deliberately not an error

  • A relations[].to that does not resolve. The record is accepted; the refs_resolved badge is withheld. A Card may legitimately be published before the Eval it cites.
  • A harness or metric the registry does not know, including a run’s metrics key. Accepted; the harness_registered / metric_registered badges are withheld. The hub does not gate on vocabulary.
  • An attachments[].sha256 that has not been uploaded. This is not a validation failure but a state conflict, 409 attachment_missing, raised by the store, because only the store knows.
  • An Eval header with zero runs, of any eval_kind. A header is complete on its own; its runs are written afterwards.

§Output

Vec<ErrorEntry { path, code, hint }>, empty on success. path is a JSON pointer into the submitted value (/results/0/metric; the root is ""). hint is prose and not part of the contract.

Enums§

RelationTarget
Where a relation points.

Constants§

MAX_RUN_ID_BYTES
Longest run_id accepted, in bytes of UTF-8.

Functions§

is_valid_id
Whether s has the form {ns}/{name}: exactly one /, both parts non-empty, each starting with [a-z0-9] and continuing with [a-z0-9._-].
is_valid_run_id
Whether s is a valid run_id: non-empty, no /, at most MAX_RUN_ID_BYTES bytes of UTF-8. Nothing else is imposed (see evalhub_schema::run); every run_id a 0.1.x producer wrote passes.
parse_relation_target
Parse a relations[].to value. None when it has none of the accepted forms.
run
Validate one run of an Eval, written under run_id (the path segment of PUT /evals/{ns}/{name}/runs/{run_id}, or the element’s own run_id in a batch). Returns every failure, sorted by (path, code); empty when the run is accepted.
validate
Validate a record of kind and return every failure, sorted by (path, code); empty when the record is accepted.