Expand description
Validation: everything the hub checks before it stores a record.
Validation runs in two passes and collects every failure from both into one list, so a producer sees all problems in one round trip.
§Pass 1: structural
The record is checked against the JSON Schema for its declared schema
(evalhub.card/1.0 or evalhub.eval/1.0) using jsonschema with draft
2020-12. The core is closed, so an unknown key anywhere outside ext is
a schema error. In the same pass every number is inspected: an integer
outside ±2^53 is number_too_large.
The schema is the generated one from evalhub_schema (the same
generator output that is committed under schemas/), built once per
process. The validator is built in offline mode: every $ref in the
schema is local (#/$defs/…), and a reference that would need fetching
is a build error, not a fetch.
§Pass 2: semantic
Rules the schema language cannot express:
| Rule | Code |
|---|---|
results non-empty ⇒ counts present | counts_missing |
counts.attempted >= completed + failed + skipped + errored | counts_inconsistent |
every results[].samples_ref, runs[].calls, runs[].artifacts[] is an attachments[].path | attachment_ref_unknown |
attachments[].path unique, relative, contains no .. | attachment_path_invalid |
results[].metric matches {ns}/{name} | metric_id_invalid |
relations[].type matches {ns}/{name} | schema |
relations[].to is {ns}/{name}@{seq}, external:…, or hf:… | schema |
The semantic pass runs on whatever is present even when the structural
pass failed, so a producer sees both kinds of problem at once. Errors
are sorted by (path, code); the order is deterministic.
§What is deliberately not an error
- A
relations[].tothat does not resolve. The record is accepted; therefs_resolvedbadge is withheld. A Card may legitimately be published before the Eval it cites. - A
harnessormetricthe registry does not know. Accepted; theharness_registered/metric_registeredbadges are withheld. The hub does not gate on vocabulary. - An
attachments[].sha256that has not been uploaded. This is not a validation failure but a state conflict,409 attachment_missing, raised by the store, because only the store knows.
§Output
Vec<Error { path, code, hint }>, empty on success. path is a JSON
pointer into the submitted record (/results/0/metric; the root is
""). hint is prose and not part of the contract.
Enums§
- Relation
Target - Where a relation points.
Functions§
- is_
valid_ id - Whether
shas the form{ns}/{name}: exactly one/, both parts non-empty, each starting with[a-z0-9]and continuing with[a-z0-9._-]. - parse_
relation_ target - Parse a
relations[].tovalue.Nonewhen it has none of the accepted forms. - validate
- Run both passes and return every failure, sorted by
(path, code).