Skip to main content

Module corpus_export

Module corpus_export 

Source
Expand description

Provenance-tagged training-corpus export (ticket .kranz/tickets/training-corpus-export.md, KRZ-332 — the governance evidence layer’s flywheel feed): one JSONL stream joining the three judged-artefact sources a local model can learn from —

  1. Validated worker tracescrate::trace_export’s instruction pairs (a run qualifies only as validation-PASSED: Role::Worker, RunResult::Pass, feature Complete inside a Complete milestone), now carrying the provenance tags. A failed or unvalidated session is excluded by CONSTRUCTION — the selection fold is reused, not reimplemented, so the corpus can never widen past the trace export’s gate.
  2. Divergence records — each divergence.noted comparison (every candidate ref verbatim: run id, branch, backend, tree hash) paired with its divergence.resolved judgement (selected index or none, reason, decider). These are the consent/judgement pairs: what the pool produced, and which side the human picked and why.
  3. Escalation-ledger records — the flight-surgeon fold (crate::escalation_metrics) as labeled human-judgment examples: grant parks (ask, decision, latency), steers, and milestone blocks the OPERATOR lifted.

Every record carries provenance refs that RESOLVE via the provenance replay (crate::provenance::provenance_chain) — they are taken FROM the replay, not recomputed beside it: the trace’s backend is the replay’s config-at-seq derivation, the gate-chain refs are the replay’s ladder seqs, the divergence seqs are the replay’s ledger seqs, and an escalation’s decision seq joins the replay’s human-decision chain. A consumer can therefore walk any record back to the mission, the session, and the gates that vouched for it without re-deriving anything.

WHY a new export-corpus command rather than extending export-traces: the trace export’s line shape IS its consumer contract (one instruction-pair object per line), and the two new sources are not instruction pairs — a consent judgement or an escalation decision has no instruction/response. Widening export-traces would either break that contract or force a tagged union onto a command whose name promises traces. A separate command keeps export-traces byte-stable and gives the corpus one stream, one ordering rule, one determinism rule.

Determinism, in the substrate’s own discipline (crate::trace_export, crate::provenance): the export is a pure function of the event log — no clock is consulted (the only time-derived value, a grant/block latency, is a difference of RECORDED timestamps), no hashed map is iterated (trace selection walks state.runs, a BTreeMap; every other fold keeps log order in Vecs), and serialization is struct-order stable. Same log → byte-identical JSONL.

WHY the ordering key is what it is: records group by source in a fixed order (worker traces, then divergences, then escalations), stable within each group — run id for traces (the BTreeMap order), the noted event’s seq for divergences, the ask event’s seq for escalations. Each within-group key is already total over one log, while a merged seq ordering would interleave sources for no gain: a corpus consumer filters by source anyway. Across missions (kranz export-corpus --all), mission ids sort (the crate::paths::MissionPaths::list_missions order), then the same grouping applies per mission.

Structs§

DivergenceRecord
One divergence.noted, paired with its resolution: the comparison record (every candidate ref verbatim — the run ids and branches the judgement chose between) plus the judgement that landed, or null while the unit awaits one. diverged: false is the agreement record: logged, never trusted — it exports like any other noted, the consumer decides what to learn from it.
DivergenceResolution
The judgement half of a divergence record: which candidate was selected (an index into the record’s candidates; null = judged-and-abandoned, serialized explicitly so it cannot be confused with a real index), why, by whom, and at which event seq (the join key into the replay’s divergence ledger).
EscalationRecord
One escalation as a labeled human-judgment example: what was asked, what was decided, how long the decision took, and the seq refs that join it to the replay. Grant/block rows mirror the ledger’s vocabulary (approved / denied: <reason> / pending); a block’s decision is unblocked: <reason>. Pending rows export with a null decision seq — the ask is real log data the consumer may want, the missing label is visible rather than silently dropped.
GateChainRef
One gate-ladder link referenced by a worker-trace record: identity, surface, and verdict of one gate.result, pinned by its event seq — the join key into the provenance replay’s ladder (crate::provenance::GateLink).
WorkerTraceRecord
A validation-PASSED worker trace: the crate::trace_export::InstructionPair fields verbatim (so a corpus line is a strict superset of an export-traces line) plus the provenance tags the ticket demands — the backend the session ran on and the mission’s gate ladder by seq.

Enums§

CorpusRecord
One corpus line: a tagged union over the three sources (module docs). The source tag is the consumer’s filter key.
EscalationKind
Which kind of escalation one record labels (the ledger’s grant/steer set plus blocks — a milestone block the operator lifted is a human judgement the ledger only counts, and the corpus names it).

Functions§

export_corpus
Derive one mission’s corpus records from its event log. events must be that mission’s log (the crate::reducer::fold discipline, same as export-traces); mission_dir anchors the provenance replay’s artefact probes. Fallible exactly where its two folds are: a log the reducer rejects, or a config.changed patch the replay cannot merge — both are corruption, never a corpus gap.
to_jsonl
Render corpus records as JSONL: one compact JSON object per line, each terminated by \n. Pure function of its input — the crate::trace_export::to_jsonl discipline — so it is byte-identical across repeated calls on the same records.