Module candidate 

A proposed harness change, and the decision about it.

This is the gate docs/SELF-IMPROVEMENT-RESEARCH.md §13.3 specifies, and it is pure on purpose: the arms are run elsewhere, and what arrives here is two sets of RunStats plus the prediction that was made before either was measured. Getting this wrong is silent — a bad rule that scores well ships and rides in every future prompt — so it is the part that gets unit tests rather than a live trial.

The shape

A candidate carries a falsifiable prediction (AHE’s decision observability): the metric it claims to move and the direction. Without one, a proposal cannot be refuted by the next measurement, and “harness updating is not harness benefit” is what follows — agents modifying themselves with no corresponding gain.

Why paired, and why a holdout

Episodes differ from each other far more than arms differ from each other, so an unpaired comparison measures which episodes landed in which arm. Pairing by episode removes that. And selecting among candidates on the same episodes that justify the winner is a multiple-comparisons trap: the more candidates, the better the winner looks and the less of it is real. So the corpus is split deterministically, selection happens on one slice, and the winner is confirmed on a slice never used for selection.

Why counts rather than a significance test

Deliberate. With a few dozen episodes the noise is the model’s sampling, not the measurement, and the answer to sampling noise is repetition (--runs k, pass^k) rather than a p-value over one sample. A test here would put a number on the wrong uncertainty and read as rigour. The raw win/loss/tie counts are reported instead, so a human reading a proposal sees what the decision was made from.

Structs

Judgement

The full result of grading a candidate, kept whole so a proposal records what it was decided from rather than only the verdict.

Pair

One episode measured in both arms. Paired by episode, which is a replay corpus id — a session id, or an eval case id.

Prediction

The claim a candidate is judged against, made before the measurement.

Tally

How one slice of the corpus came out.

Enums

ChangeClass

What kind of change this is, which decides how far it can get without a person. See §13.2–13.3 of the research.

Disposition

What the gate decided, and why in words a human can check.

Metric

What a candidate claims it will do.

Constants

MIN_HOLDOUT_PAIRS

MIN_SELECTION_PAIRS

Below this many paired episodes in a slice, a difference is not evidence.

WORK_FLOOR

How far work may fall before a gain is treated as bought rather than earned. Some drop is legitimate — a change that stops a redundant re-read does less work and is better for it — so this is a cliff, not a ratchet.

Functions

is_holdout

Split an episode into selection or holdout, deterministically.

judge

Grade a candidate against its own prediction.

judge_with

The same gate over anything that can name an episode and produce a cost.

pair_arms

Pair two arms by episode id, dropping anything that ran in only one.