Skip to main content

Module validate

Module validate 

Source
Expand description

Validation: everything the hub checks before it stores a record.

Validation runs in two passes and collects every failure from both into one list, so a producer sees all problems in one round trip.

§Pass 1: structural

The record is checked against the JSON Schema for its declared schema (evalhub.card/1.0 or evalhub.eval/1.0) using jsonschema with draft 2020-12. The core is closed, so an unknown key anywhere outside ext is a schema error. In the same pass every number is inspected: an integer outside ±2^53 is number_too_large.

The schema is the generated one from evalhub_schema (the same generator output that is committed under schemas/), built once per process. The validator is built in offline mode: every $ref in the schema is local (#/$defs/…), and a reference that would need fetching is a build error, not a fetch.

§Pass 2: semantic

Rules the schema language cannot express:

RuleCode
results non-empty ⇒ counts presentcounts_missing
counts.attempted >= completed + failed + skipped + erroredcounts_inconsistent
every results[].samples_ref, runs[].calls, runs[].artifacts[] is an attachments[].pathattachment_ref_unknown
attachments[].path unique, relative, contains no ..attachment_path_invalid
results[].metric matches {ns}/{name}metric_id_invalid
relations[].type matches {ns}/{name}schema
relations[].to is {ns}/{name}@{seq}, external:…, or hf:…schema

The semantic pass runs on whatever is present even when the structural pass failed, so a producer sees both kinds of problem at once. Errors are sorted by (path, code); the order is deterministic.

§What is deliberately not an error

  • A relations[].to that does not resolve. The record is accepted; the refs_resolved badge is withheld. A Card may legitimately be published before the Eval it cites.
  • A harness or metric the registry does not know. Accepted; the harness_registered / metric_registered badges are withheld. The hub does not gate on vocabulary.
  • An attachments[].sha256 that has not been uploaded. This is not a validation failure but a state conflict, 409 attachment_missing, raised by the store, because only the store knows.

§Output

Vec<Error { path, code, hint }>, empty on success. path is a JSON pointer into the submitted record (/results/0/metric; the root is ""). hint is prose and not part of the contract.

Enums§

RelationTarget
Where a relation points.

Functions§

is_valid_id
Whether s has the form {ns}/{name}: exactly one /, both parts non-empty, each starting with [a-z0-9] and continuing with [a-z0-9._-].
parse_relation_target
Parse a relations[].to value. None when it has none of the accepted forms.
validate
Run both passes and return every failure, sorted by (path, code).