Expand description
Paired report + non-regression gate (#237).
Every task is scored under both conditions, giving a paired sample of per-task deltas
(lean_ctx − baseline). From those we compute the mean delta, a deterministic bootstrap
95% confidence interval (fixed seed → byte-identical CI on every machine), win/tie/loss
counts and pass-rate deltas, then collapse it all into a single Verdict that drives the
CI quality gate.
Structs§
- AbReport
- The full A/B report: provenance, per-task records, aggregate stats and the verdict.
- AbStats
- Aggregate statistics over all paired records.
- Pair
Record - One task scored under both conditions, with the audit digests for each window + answer.
- Report
Config - Knobs for the statistics + gate. Defaults are deterministic and strict.
Enums§
- Verdict
- The headline conclusion of a run.
Constants§
- REPORT_
KIND - Report schema discriminator + version (also guards artifact parsing).
- REPORT_
SCHEMA_ VERSION