Module eval_ab 

Deterministic with/without output-quality eval (#232).

Proves — reproducibly and with a signature — whether putting lean-ctx in front of a model changes the quality of its answers, not just the token count. The design separates the two sources of variance:

• Context is deterministic. Both the baseline (“raw dump”) and the lean-ctx (“retrieve + compress”) window are assembled byte-for-byte reproducibly and digested.
• The model is the only stochastic part. It is pinned (temperature = 0, fixed seed) and, for CI, replaced by model::RecordedRunner replaying captured real responses, so a run is byte-identical everywhere.

The pipeline per task is: conditions::assemble → model::ModelRunner → scorers::score_task. Results become a paired report::AbReport, which a artifact::SignedAbReportV1 turns into a portable, verifiable attestation.

Modules

artifact

Signed, reproducible artifact (#238).

conditions

A/B context conditions (#235): the two context layers a task is run under.

footprint

Footprint ablation eval (#959) — proving lean-ctx’s OWN injected context earns its tokens.

judge

Real LLM-as-judge scorer for free-form QA (#611).

model

Pinned, reproducible model adapter (#234).

report

Paired report + non-regression gate (#237).

scorers

Deterministic scorers (#236): objective, reproducible scoring of a model answer.

suite

Eval suite + fixtures (#233): the deterministic task definitions an A/B run scores.

testbench

Public off-vs-on answer-quality testbench (#611).

Structs

AbRunConfig

Configuration for one A/B run.

Functions

run_ab

Runs every task in suite under both conditions through runner, scoring each answer, and assembles the paired report. The model is the only non-deterministic input.