Crate warble_eval_runner

Expand description

Warble eval runner.

Replays golden questions through a dispatched Warble agent under several tier→model bindings, scores each result set with warble-eval-compare, and produces a Pareto report (accuracy vs cost vs latency). The whole-model binding is runtime-injected via claude --model — same IR/agent, different binding, which is exactly what run_eval varies. The queryable project (connection + data) is injected via project; the agent files are installed into <project>/.claude for the run and removed afterward.

On top of that measurement, the closed loop (roadmap Phase 1.4) lives in sibling modules:

ablation — per-step tier ablation: re-dispatch the IR binding one named step at a time.
gate — CI gate: fail on accuracy regression vs a committed baseline.
context — MDL-version reverify: flag goldens whose pinned context_version went stale.
capture — capture-confirmed: mint a candidate golden from one confirmed run.

Pure helpers (extract_result, aggregate, format_pareto, golden parsing) and the closed-loop modules’ logic are unit-tested; run_eval / run_ablation additionally spawn claude and are exercised end-to-end against a live project.

Structs§

AblationConfig: Inputs for a per-step ablation sweep.
AblationPoint: One ablation measurement: step label bound to tier, scored against the goldens.
AblationReport
CaptureInput: Everything needed to mint one candidate golden case.
CaseFilter: A --tags + --sample selection over a golden file’s cases.
CaseKey: The inputs that determine a case’s result. Everything here is in the cache key; the golden’s expectation (expected/tolerance/match) deliberately is not, which is what makes re-scoring a changed expectation a cache hit.
CaseResult: A case’s aggregate result over samples repeated runs (pass-rate methodology). At samples == 1 every field is bit-identical to the pre-repeated-sampling report shape — that identity is the whole back-compat story, both for the gate’s case-level comparison and for the Pareto table.
Check
ComplianceIr: A minimal, local view of the compiled IR — just enough to resolve a trace’s component and read its locked guardrails. Deliberately decoupled from warble_claude_code::ir::WarbleIr (same pattern as mutate_change.rs’s inline golden graph): the scorer only needs a handful of fields, and a narrow local view can’t accidentally start depending on IR internals that later change.
ComplianceReport: A trace’s compliance verdict against one component’s locked guardrails. compliant is exactly “no Fail among checks” — a NotChecked guardrail never blocks compliance (it isn’t a violation, it’s a gap in scorer coverage), but it is always visible in checks, never omitted.
ComplianceTrace: One captured run: which component was dispatched, the pinned target it was bound to (if any), and its ordered tool-call/approval events. Order matters — every check below reasons about “did X happen before Y”, not just “did X happen at all”.
ConfigReport
ContextInjectionReport: Strict, non-sensitive projection of a dispatched agent’s context identity.
FlakyCase: A case that passed fully in the baseline but is inconsistent (0 < pass_rate < 1) in the candidate — reported separately from Regression because it isn’t a hard regression: the case can still pass, just not every time.
GateResult: The gate verdict: passed plus every regression and any structural notes.
Golden: A golden eval file: dataset metadata + cases with expected result sets.
GoldenCase
MonitorMetric
MonitorObservation
MonitorReport
Regression: One regression the gate found, named so a human knows exactly what got worse.
Report: A full eval report (one config per tier→model binding). Serialized by warble eval run --out and consumed as the baseline / candidate by warble eval gate.
RunConfig
SampleVerdict: One sample’s verdict, kept in CaseResult::samples_detail so a flaky case’s individual sample-by-sample results (which ones failed, and why) survive past the aggregate.
Selection: The result of applying a CaseFilter: indices into the original slice (ascending), plus a note recording selected/total so a partial (smoke) run is always visible.
StepRecommendation: The cheapest tier that holds accuracy for one step (the closed loop’s per-step verdict).
TagStat
Trace: One captured case result. The result is stored as the raw {columns, rows} JSON value (the same object capture embeds) so no Serialize is needed on the compare crate’s Table; it is deserialized back into a Table at re-score time.
TraceStore: A content-addressed store of Traces under one directory (<dir>/<key-hash>.json).
VerifyResult

Enums§

Backend: Which back-end/runtime a case was (or should be) run against. Value domain matches the dispatcher/<name> directory names exactly, so clap’s own unknown-value error already lists every back-end this crate knows the name of — but knowing the name is not the same as having an adapter for it; see resolve_adapter (crate-private) below.
CheckStatus: One guardrail’s verdict. NotChecked is not a pass — it exists so a locked guardrail the scorer doesn’t (yet) model is always visible in the report rather than silently absent (no silent caps, same principle as everywhere else in this eval suite).
Freshness: Whether a golden’s pinned context_version still matches the bound MDL.
OutputStability: A case’s output-level stability across repeated samples — independent of pass/fail. flaky only tells you the verdict flipped; a case can fail every sample with a different wrong answer each time and flaky never notices, because pass never changes. This classification is derived on read from samples/answer_dist rather than stored, so it can never drift out of sync with the data it summarizes and needs no new field or legacy-report migration.
ResultKind: Which shape a case’s expected compares against — additive so every existing (all-Table) golden stays byte-compatible: an omitted result_kind defaults to Table, the pre-existing behavior. Verdict is the +Assertive twin: the agent’s final message is a {blocks, verdict, emitted, verified} envelope (see dispatcher/claude-code-cli’s assertion output contract), not a {columns, rows} table, and one of its fields is projected down to expected’s scalar shape before the usual comparator runs — see project_verdict_field.
Sample: How to sub-select from the (tag-filtered) golden cases. Parsed from --sample.
TraceEvent: One event in a trace, tagged by t — the same shape a live captured stream would emit.

Functions§

aggregate: Aggregate per-case results for one model binding into a config report.
build_candidate_yaml: Render one confirmed run as a golden-shaped case block (a single - … list item), ready to append under a candidates file’s cases:. Fails if the result carries no {columns, rows} or the match mode is unrecognized — a candidate that wouldn’t parse as a golden is not worth writing.
candidates_header: The candidates staging file header — a golden-shaped document so accepted candidates can be moved straight into a golden set. Written once when the file is first created.
classify: Classify a pin against a freshly computed MDL SHA.
compute_mdl_sha: Compute a single stable SHA over all MDL files: hash each file’s content (git blob SHA), build a sorted <relpath> <sha> manifest, and hash the manifest. Any content or path change moves it.
effective_record_answers: Whether a run should record each sample’s answer value (CaseResult::answer_dist), given the explicit --record-answers flag and the resolved --samples count.
extract_result: Extract the {columns, rows} object from an agent’s free-form final text: prefer a fenced code block, else the whole text; then take the first { … last } and parse it as a table.
extract_result_json: Like extract_result but returns the raw {columns, rows} JSON object (kept serializable so warble eval capture can embed it verbatim as a golden’s expected).
extract_verdict_json: Extract a {blocks, verdict, emitted, verified} verdict envelope from an agent’s free-form final text — the +Assertive twin of extract_result_json. Same fence-stripping and outermost-brace slicing, but the acceptance guard checks for verdict or blocks instead of rows, since a verdict envelope carries neither of the table’s fields. Rejects anything else (including a plain {columns,rows} table) so a verdict case never accidentally scores a table-shaped answer.
format_ablation: Render the per-step ablation as a table: baseline first, then each (step, tier) with Δacc / Δcost, followed by the per-step recommendation.
format_compliance: Render a ComplianceReport in the same reporting voice as format_gate — a short banner, a line per check, and a final compliant/non-compliant verdict naming exactly what failed.
format_gate: Render the gate verdict for humans / CI logs.
format_monitor_report
format_pareto: Render the Pareto table (accuracy vs cost vs latency, with per-tag accuracy).
parse_context_version: Split a context_version (<dataset>@<pin>) into its dataset prefix and pin. With no @, the whole value is the pin and there is no dataset.
project_verdict_field: Project one field of a verdict envelope down to the 1x1 scalar Table a case’s expected compares against. "fresh" reads verdict.fresh; "severity" reads the breach severity off the envelope’s status block (falling back to verdict.severity, in case a future envelope carries it there instead). Fails closed: an absent, wrong-typed, or unrecognized field returns None — the case then fails outright rather than silently comparing something else.
rescore: Re-score a cached trace against a golden case’s current expectation — the 0-LLM path. Dispatches on the case’s crate::ResultKind via score_value just like a fresh run does, so a Table case’s cached {columns,rows} and a Verdict case’s cached envelope both re-project the same way. Returns None if the cached result doesn’t have the shape the case’s kind expects (a corrupt entry, or a case whose result_kind changed since the trace was captured; the caller then re-runs the case). This is what makes “only the golden’s expected changed” cost zero LLM calls: the result is reused, only the comparison is redone.
run_ablation: Run the per-step ablation: baseline (all steps at base_tier) plus, for each named step, one re-dispatch per swept tier. Streams progress to stderr and returns the aggregated report.
run_eval
run_gate: Gate a candidate report against a baseline. A metric regresses when current < baseline - tolerance. Configs are matched by their model label; a case regresses when it passed in the baseline but fails or is absent in the candidate.
score_compliance: Score trace against ir’s declared guardrails for the trace’s component. Pure: no file I/O, no process spawn, no model call — the CLI (run_eval_compliance) does the reading, this function only reasons over already-parsed data, which is what makes it trivially unit-testable.
score_monitor_pair: Score the one currently-live freshness scenario. Other injection kinds deliberately remain a follow-up: only stopped_updates has deterministic freshness labels in the manifest.
score_value: Score a raw result value against a case’s expectation, dispatching on ResultKind so the fresh-run path (run_case) and the cache re-score path (cache::rescore) project the same way from the same stored shape: a Table value for ResultKind::Table, a verdict envelope for ResultKind::Verdict. Returns None when result_value doesn’t have the shape the case’s kind expects — the caller treats that as a hard fail, never a false pass.
select_cases: Apply filter to cases: tag-filter first (union over --tags), then --sample. Deterministic and order-preserving so re-runs are comparable.
stamp_context_version: Rewrite a golden YAML’s context_version: line to <dataset>@<sha>, preserving the dataset prefix when the golden already had one (else falling back to fallback_dataset). Line-based so comments and formatting survive. Returns the new document text.
verify_context: Verify a golden’s context_version against the bound project’s current MDL SHA.