Expand description
Dataset identity attestation and its re-derivation from supplied buffers.
Implemented by plan 02-06. Dataset identity attestation and its re-derivation from supplied buffers.
§Contract: contrastive-pair-protocol-v1.yaml (equation dataset_attestation)
A [DatasetAttestation] carries profile, schema version, label map, per-split JSONL
SHA-256, per-split per-class counts, normalization version, the cross-split
exclusion-record digest, and the dataset fingerprint. from_attested_bytes re-derives
every one of those from the buffers the caller supplied and fails typed on the first
disagreement — an attestation that is merely quoted back proves nothing about the
bytes in hand.
§The threat this closes
Row-level checks are not enough. A consumer pointed at an output directory whose
train.jsonl came from one preparation and validation.jsonl from another would pass
every row-level gate: each file is individually well-formed, each row carries the right
role, each class count is internally consistent. What is broken is split identity —
the two files do not describe one dataset — and nothing a row can say detects it. The
attested per-split digests plus the whole-dataset fingerprint are what turn that mixed
directory into a typed ContrastiveDataError::SplitHashMismatch or
ContrastiveDataError::FingerprintMismatch instead of a silent success.
§Order is part of the guarantee
The ladder runs: parse the attestation -> schema version -> normalization version -> profile -> role set -> per-split SHA-256 BEFORE the buffer is parsed -> the ordinary ingest gate ladder (which is where per-class counts are checked) -> exclusion digest -> dataset fingerprint. The split-hash check precedes parsing deliberately: a corrupted buffer must be reported as “these are not the bytes you attested”, not as “row 4 is malformed”. The second diagnosis sends a reader looking for a data-quality problem in a file that is simply the wrong file.
No value of type PreparedDataset<P> — and therefore no Split<R> accessor — exists
until every comparison has passed. Exposure-then-validate would let a caller read rows
out of a dataset that is about to be rejected.
§There is no schema-version-1 migration, deliberately
See [SUPPORTED_DATASET_ATTESTATION_SCHEMA_VERSIONS].
Structs§
- Dataset
Attestation - The identity a prepared dataset attests to.
- Split
Attestation - What one split’s bytes must reproduce.
Constants§
- DATASET_
ATTESTATION_ SCHEMA_ VERSION - The attestation schema version this build WRITES.
- SUPPORTED_
DATASET_ ATTESTATION_ SCHEMA_ VERSIONS - Every attestation schema version this build ACCEPTS.
Traits§
- Attested
Profile - A profile that can be attested. Implemented only by
CanonicalandCompatibility.