Module ab 

Paired A/B comparator: two arms — each an (ArmSpec) engine on a backbone — over the same task and the same OutcomeContract.

The original claim was “CAR + gpt-5.5 codes as well as Codex + gpt-5.5, and here is where its harness wins.” That is only meaningful when both arms run the same model — ALE’s finding is that backbone choice is ~3× the spread of harness choice, so native-on-Haiku vs Codex-on-gpt-5.5 measures the model, not the runtime.

Arms therefore carry their own backbone, which lets one comparator express the whole family (see docs/proposals/coder-ab-value-surface.md): the same-backbone harness delta, the cross-tier diagonal (native@gpt-5.4 vs codex@gpt-5.5 — what the runtime is worth in model tiers), and ablation (native vs native with a mechanism off, the only shape that isolates a single mechanism).

The fairness rule is now checked rather than asserted (AbReport::same_backbone). It previously could not be: the report held one operator-supplied backbone string while the pin reached only the native loop, so the external arm ran on whatever its CLI was configured for.

Design mirrors bench/car_bench/ale’s orchestrator, ported to Rust and to the in-process coder path:

• Honest denominator. An arm that never actually ran (worktree/setup/ inference-transport failure) is infra_failed and excluded — never scored as a task-0. The paired stats are computed over the tasks both arms scored, exactly like compare_pair’s treatment ∩ control intersection.
• Paired significance. McNemar’s test over the discordant pairs (treatment-pass/control-fail vs treatment-fail/control-pass) — the right test for two harnesses judged on the same items, and the regression gate Slice 4 re-runs to decide whether an applied change was a real improvement or noise.
• Cost axis. Per-arm mean cost and cost-per-pass, so “as good” can be qualified by “and cheaper/dearer.”

The execution seam is injected (AbArmRunner) — the same injected-closure philosophy as car-builder’s generate seam and car-verify::cwm’s EffectModel — so the statistics core is unit-testable with scripted arms and no live inference. The live runner (real run_native_loop / run_external_loop in fresh worktrees) is wired by the car coder-ab CLI.

Structs

AbCell

Both arms’ outcomes on one task — the paired unit.

AbReport

The full report: every paired cell plus the aggregate stats. Serializes to the timestamped JSON the car coder-ab CLI writes to bench/results.

AbTask

One coding task both arms attempt, verified against the identical contract. The corpus is a JSONL of these — curated from CAR’s own merged PRs / closed issues (revert the fix, keep the PR’s tests as the contract, have both arms re-derive it).

ArmOutcome

A single arm’s result on a single task, normalized so both the native LoopOutcome and the external CLI’s result map onto one shape.

ArmSpec

A fully-specified arm: which engine, on which backbone.

PairedStats

Aggregate paired statistics over the scorable intersection.

RoundAttribution

Attribution over one A/B round’s treatment-arm losses.

Enums

ArmEngine

Which engine an arm drives.

Traits

AbArmRunner

The injected execution seam: run one arm on one task, in its own fresh worktree, verifying against task.contract. Never panics — an arm that can’t be attempted returns ArmOutcome::infra. Scripted in tests; the live impl (real run_native_loop / run_external_loop) is wired by the CLI.

Functions

attribute_round

Attribute the treatment-arm losses in report. read_events(path) returns the event-log JSONL for a transcript path (live: read the file; tests: an in-memory map). Only paired-scorable native losses are attributed — an infra failure is not a harness lesson. Interventions from a layer that defers elsewhere (is_evolution_actionable false) still mark a loss as addressed but are dropped from the fixer feed.

is_evolution_actionable

Whether a diagnosed layer is one the Evolution Agent can act on autonomously. ProceduralSkill defers to CAR’s separate skill-distillation path, so it is not fed to the harness-evolution fixer here.

run_ab_suite

Run the full paired suite: each task through both arms, against the same contract, folded into an AbReport.

run_ab_suite_resumable

Resumable, incrementally-observable run_ab_suite. A long real-corpus run is expensive, so this makes it interruptible and iterative: