Module testbench 

Public off-vs-on answer-quality testbench (#611).

One command runs every pinned repo in a lockfile::TestbenchLock under both arms — off (Condition::Baseline, a raw file dump) and on (Condition::LeanCtx, retrieve + compress) — at an identical token budget, then scores each answer:

• free-form QA → real LlmJudge (the pinned model grades correctness),
• code → CodeScorer, the SWE-style test-oracle (apply the answer, run the repo’s own test command, pass = exit 0).

Results reuse the AbReport statistics + verdict per repo and are aggregated into a TestbenchReport whose determinism_digest excludes wall-clock time, so a recorded subset is byte-identical everywhere and can gate CI. findings renders the human FINDINGS.md (per-repo tokens / turns / walltime / quality) and the honest machine-readable regressions file.

Modules

clone

Repo materialization for the testbench (#611).

findings

Human + machine outputs for a testbench run (#611).

lockfile

Pinned-repo lockfile for the public off-vs-on testbench (#611).

recording

Deterministic recording builder for the committed testbench subset (#611).

Structs

RepoReport

One repo’s outcome: the reused paired AbReport plus informational wall-clock time per arm (deliberately outside the determinism digest).

TestbenchConfig

Knobs for a testbench run (token budget + the per-repo report/gate config).

TestbenchReport

The aggregate attestation written by a run.

Constants

TESTBENCH_REPORT_KIND

Report schema discriminator.

Functions

run_testbench

Runs every repo in lock under both arms through runner, returning the aggregate. runner answers and judges (judge requests have distinct content → distinct keys).