Crate warble_eval_runner

Expand description

Warble eval runner.

Replays golden questions through a dispatched Warble agent under several tier→model bindings, scores each result set with warble-eval-compare, and produces a Pareto report (accuracy vs cost vs latency). The whole-model binding is runtime-injected via claude --model — same IR/agent, different binding, which is exactly what run_eval varies. The queryable project (connection + data) is injected via project; the agent files are installed into <project>/.claude for the run and removed afterward.

On top of that measurement, the closed loop (roadmap Phase 1.4) lives in sibling modules:

ablation — per-step tier ablation: re-dispatch the IR binding one named step at a time.
gate — CI gate: fail on accuracy regression vs a committed baseline.
context — MDL-version reverify: flag goldens whose pinned context_version went stale.
capture — capture-confirmed: mint a candidate golden from one confirmed run.

Pure helpers (extract_result, aggregate, format_pareto, golden parsing) and the closed-loop modules’ logic are unit-tested; run_eval / run_ablation additionally spawn claude and are exercised end-to-end against a live project.

Structs§

AblationConfig: Inputs for a per-step ablation sweep.
AblationPoint: One ablation measurement: step label bound to tier, scored against the goldens.
AblationReport
CaptureInput: Everything needed to mint one candidate golden case.
CaseFilter: A --tags + --sample selection over a golden file’s cases.
CaseKey: The inputs that determine a case’s result. Everything here is in the cache key; the golden’s expectation (expected/tolerance/match) deliberately is not, which is what makes re-scoring a changed expectation a cache hit.
CaseResult: A case’s aggregate result over samples repeated runs (pass-rate methodology). At samples == 1 every field is bit-identical to the pre-repeated-sampling report shape — that identity is the whole back-compat story, both for the gate’s case-level comparison and for the Pareto table.
Check
ComplianceIr: A minimal, local view of the compiled IR — just enough to resolve a trace’s component and read its locked guardrails. Deliberately decoupled from warble_claude_code::ir::WarbleIr (same pattern as mutate_change.rs’s inline golden graph): the scorer only needs a handful of fields, and a narrow local view can’t accidentally start depending on IR internals that later change.
ComplianceReport: A trace’s compliance verdict against one component’s locked guardrails. compliant is exactly “no Fail among checks” — a NotChecked guardrail never blocks compliance (it isn’t a violation, it’s a gap in scorer coverage), but it is always visible in checks, never omitted.
ComplianceTrace: One captured run: which component was dispatched, the pinned target it was bound to (if any), and its ordered tool-call/approval events. Order matters — every check below reasons about “did X happen before Y”, not just “did X happen at all”.
ConfigReport
FlakyCase: A case that passed fully in the baseline but is inconsistent (0 < pass_rate < 1) in the candidate — reported separately from Regression because it isn’t a hard regression: the case can still pass, just not every time.
GateResult: The gate verdict: passed plus every regression and any structural notes.
Golden: A golden eval file: dataset metadata + cases with expected result sets.
GoldenCase
Regression: One regression the gate found, named so a human knows exactly what got worse.
Report: A full eval report (one config per tier→model binding). Serialized by warble eval run --out and consumed as the baseline / candidate by warble eval gate.
RunConfig
SampleVerdict: One sample’s verdict, kept in CaseResult::samples_detail so a flaky case’s individual sample-by-sample results (which ones failed, and why) survive past the aggregate.
Selection: The result of applying a CaseFilter: indices into the original slice (ascending), plus a note recording selected/total so a partial (smoke) run is always visible.
StepRecommendation: The cheapest tier that holds accuracy for one step (the closed loop’s per-step verdict).
TagStat
Trace: One captured case result. The result is stored as the raw {columns, rows} JSON value (the same object capture embeds) so no Serialize is needed on the compare crate’s Table; it is deserialized back into a Table at re-score time.
TraceStore: A content-addressed store of Traces under one directory (<dir>/<key-hash>.json).
VerifyResult

Enums§

CheckStatus: One guardrail’s verdict. NotChecked is not a pass — it exists so a locked guardrail the scorer doesn’t (yet) model is always visible in the report rather than silently absent (no silent caps, same principle as everywhere else in this eval suite).
Freshness: Whether a golden’s pinned context_version still matches the bound MDL.
ResultKind: Which shape a case’s expected compares against — additive so every existing (all-Table) golden stays byte-compatible: an omitted result_kind defaults to Table, the pre-existing behavior. Verdict is the +Assertive twin: the agent’s final message is a {blocks, verdict, emitted, verified} envelope (see dispatcher/claude-code-cli’s assertion output contract), not a {columns, rows} table, and one of its fields is projected down to expected’s scalar shape before the usual comparator runs — see project_verdict_field.
Sample: How to sub-select from the (tag-filtered) golden cases. Parsed from --sample.
TraceEvent: One event in a trace, tagged by t — the same shape a live captured stream would emit.

Functions§

aggregate: Aggregate per-case results for one model binding into a config report.
build_candidate_yaml: Render one confirmed run as a golden-shaped case block (a single - … list item), ready to append under a candidates file’s cases:. Fails if the result carries no {columns, rows} or the match mode is unrecognized — a candidate that wouldn’t parse as a golden is not worth writing.
candidates_header: The candidates staging file header — a golden-shaped document so accepted candidates can be moved straight into a golden set. Written once when the file is first created.
classify: Classify a pin against a freshly computed MDL SHA.
compute_mdl_sha: Compute a single stable SHA over all MDL files: hash each file’s content (git blob SHA), build a sorted <relpath> <sha> manifest, and hash the manifest. Any content or path change moves it.
extract_result: Extract the {columns, rows} object from an agent’s free-form final text: prefer a fenced code block, else the whole text; then take the first { … last } and parse it as a table.
extract_result_json: Like extract_result but returns the raw {columns, rows} JSON object (kept serializable so warble eval capture can embed it verbatim as a golden’s expected).
extract_verdict_json: Extract a {blocks, verdict, emitted, verified} verdict envelope from an agent’s free-form final text — the +Assertive twin of extract_result_json. Same fence-stripping and outermost-brace slicing, but the acceptance guard checks for verdict or blocks instead of rows, since a verdict envelope carries neither of the table’s fields. Rejects anything else (including a plain {columns,rows} table) so a verdict case never accidentally scores a table-shaped answer.
format_ablation: Render the per-step ablation as a table: baseline first, then each (step, tier) with Δacc / Δcost, followed by the per-step recommendation.
format_compliance: Render a ComplianceReport in the same reporting voice as format_gate — a short banner, a line per check, and a final compliant/non-compliant verdict naming exactly what failed.
format_gate: Render the gate verdict for humans / CI logs.
format_pareto: Render the Pareto table (accuracy vs cost vs latency, with per-tag accuracy).
parse_context_version: Split a context_version (<dataset>@<pin>) into its dataset prefix and pin. With no @, the whole value is the pin and there is no dataset.
project_verdict_field: Project one field of a verdict envelope down to the 1x1 scalar Table a case’s expected compares against. "fresh" reads verdict.fresh; "severity" reads the breach severity off the envelope’s status block (falling back to verdict.severity, in case a future envelope carries it there instead). Fails closed: an absent, wrong-typed, or unrecognized field returns None — the case then fails outright rather than silently comparing something else.
rescore: Re-score a cached trace against a golden case’s current expectation — the 0-LLM path. Dispatches on the case’s crate::ResultKind via score_value just like a fresh run does, so a Table case’s cached {columns,rows} and a Verdict case’s cached envelope both re-project the same way. Returns None if the cached result doesn’t have the shape the case’s kind expects (a corrupt entry, or a case whose result_kind changed since the trace was captured; the caller then re-runs the case). This is what makes “only the golden’s expected changed” cost zero LLM calls: the result is reused, only the comparison is redone.
run_ablation: Run the per-step ablation: baseline (all steps at base_tier) plus, for each named step, one re-dispatch per swept tier. Streams progress to stderr and returns the aggregated report.
run_eval: Load the golden file, install the agent, run every case under every model binding, and return the aggregated report. Progress is streamed to stderr. The agent install is cleaned up on return (including on error) via the InstalledAgents drop guard.
run_gate: Gate a candidate report against a baseline. A metric regresses when current < baseline - tolerance. Configs are matched by their model label; a case regresses when it passed in the baseline but fails or is absent in the candidate.
score_compliance: Score trace against ir’s declared guardrails for the trace’s component. Pure: no file I/O, no process spawn, no model call — the CLI (run_eval_compliance) does the reading, this function only reasons over already-parsed data, which is what makes it trivially unit-testable.
score_value: Score a raw result value against a case’s expectation, dispatching on ResultKind so the fresh-run path (run_case) and the cache re-score path (cache::rescore) project the same way from the same stored shape: a Table value for ResultKind::Table, a verdict envelope for ResultKind::Verdict. Returns None when result_value doesn’t have the shape the case’s kind expects — the caller treats that as a hard fail, never a false pass.
select_cases: Apply filter to cases: tag-filter first (union over --tags), then --sample. Deterministic and order-preserving so re-runs are comparable.
stamp_context_version: Rewrite a golden YAML’s context_version: line to <dataset>@<sha>, preserving the dataset prefix when the golden already had one (else falling back to fallback_dataset). Line-based so comments and formatting survive. Returns the new document text.
verify_context: Verify a golden’s context_version against the bound project’s current MDL SHA.