newt-eval — end-to-end evaluation framework for the newt worker

newt-eval is the scorecard for newt worker. It answers:

Does this version of the worker actually produce useful patches?

It is also the dogfood substrate for the drake-arbiter scoring pattern — the same test-case + evaluators + scorecard shape the arbiter uses to grade swarm candidates.

Two modes

Mock mode is fully deterministic — every case ships with a canned diff that the wiremock returns verbatim. The evaluators then verify the worker did the right thing with that response (parsed it, applied it to the workspace, captured the resulting git diff, returned a well-formed TaskReply).

Live mode is for comparing models / catching regressions: the worker runs against your real Ollama and the same evaluators grade whatever the model returns. Weak models will legitimately fail — that's the point.

Running

# Mock e2e (runs in CI):
cargo test -p newt-eval --test mock_e2e

# List the bundled cases:
cargo run -q --bin newt-eval -- list-cases

# Live: all cases against discovered Ollama:
just eval

# Live: one case, specific model:
just eval --case 001 --model llama3.1:8b

# Coder mode + raised timeout for a slower local model:
newt-eval run --mode live --case 008 --coder --worker-timeout-ms 180000

Coder mode and slow models

• --coder spawns the worker with the newt-coder plugin (whole-file emit + server-side diff normalization). Use it for local Ollama coder models that otherwise trip failure mode T0b (invented hunk context).
• --worker-timeout-ms <ms> (env NEWT_EVAL_WORKER_TIMEOUT_MS, default 60000) is the per-case wall-clock budget. A model slower than this is scored dispatch_error even if it would have produced correct output — raise it (e.g. 180000) when evaluating slower models so the scorecard measures capability, not just latency.

Exit codes from newt-eval run:

• 0 — every case passed every evaluator
• 2 — ran cleanly but at least one case failed
• 1 — hard error (worker not found, cases dir missing, etc.)

The five evaluators

Language-specific evaluators auto-skip non-Rust cases with a clear note. tests_pass also skips Rust cases that have no #[test] anywhere so trivial cases don't pay the cargo-test bill.

The output_matches result oracle (#957)

Beyond the diff/pattern checks, output_matches grades what the program prints — it runs the graded program and compares its stdout to a declared expected_output. It is opt-in (add "output_matches" to a case's evaluators; it is never in the defaults) and passes with a skip note when a case declares no expected_output / [output_match].

A case configures it with a top-level expected_output and an [output_match] table:

evaluators = ["output_matches"]

# What the program's stdout must equal (after the normalize pipeline below).
expected_output = "42"

[output_match]
# argv to run in the graded worktree. argv[0] is the program/interpreter.
run = ["python3", "solution.py"]
# Hard wall-clock budget; the process is killed on expiry and reported timed_out.
timeout_ms = 10000
# Ordered normalization strategies (#959), applied to BOTH sides before compare.
# Empty list = exact byte match.
normalize = ["regex_extract", "numeric_tolerance"]
# For regex_extract: reduce each side to capture group 1 (pull the answer out of
# surrounding noise). Program may print "computing...\nResult: 42.0\n".
extract_pattern = 'Result:\s*([\d.]+)'
# For numeric_tolerance: compare as numbers within epsilon (default 1e-9).
epsilon = 0.001

Normalization strategies (pure data — new tolerance is config, not code):

Unknown strategy names, and regex_extract with a missing/invalid extract_pattern, are skipped with a warning (never fatal) — the same tolerant-load rule the language packs use.

Trust boundary + tiering. The real runner (SubprocessRunner) executes untrusted, model-produced code and does not sandbox it (no network or filesystem confinement beyond cwd; see #887). The real-subprocess tests live in tests/output_matches_real.rs, are #[ignore]d out of the per-PR unit run, and run in the weekly + release tiers, single-threaded:

cargo test -p newt-eval --test output_matches_real -- --ignored --test-threads=1

(CI: .github/workflows/output-oracle-real.yml.)

Adding a new case

A case is a directory under newt-eval/cases/NNN-name/ with two parts:

1. case.toml — metadata + mock response.
2. workspace/ — the initial filesystem state, copied verbatim into a tempdir at the start of each run.

See cases/CASE_AUTHORING.md for the full guide — the emission-contract rule and how to generate byte-accurate goldens from git diff.

# newt-eval/cases/010-your-case/case.toml
name = "010-your-case"
description = "One-line summary"
language = "rust"
difficulty = "L1"   # L1 saturated edit · L2 multi-step · L3 cross-domain
prompt = """
Multi-line instruction describing the TASK only. Do NOT specify the
response format — the system prompt owns the emission shape, and a
`case_prompt_lint` test fails on directives like "respond with a
unified diff only" (see CASE_AUTHORING.md).
"""

evaluators = [
    "diff_nonempty",
    "diff_applies",
    "rust_compiles",
    "tests_pass",
    "pattern_match",
]

# At least one of these regexes must match the captured diff.
expected_patterns = [
    "fn new_thing\\(",
]

# Mock-mode response — exactly what the wiremock will return as the
# Ollama assistant content. Must be a valid unified diff against the
# workspace/ fixture so the worker's apply_patch can take it.
[mock_response]
content = """
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,3 +1,3 @@
-old line
+new line
"""

Verify a new case before committing:

# 1. The case loads cleanly:
cargo run -q --bin newt-eval -- list-cases

# 2. The mock diff would actually apply to the fixture
#    (independent of the worker — sanity check):
cp -r newt-eval/cases/006-your-case/workspace /tmp/v
cd /tmp/v && git init -q && git add -A && git commit -q -m b
git apply --check /path/to/your.diff && echo OK

# 3. End-to-end with the worker:
cargo test -p newt-eval --test mock_e2e

Test budget

The mock e2e test runs all bundled cases in well under 30 seconds in CI once the worker binary is built. Each case spawns a worker subprocess, drives ACP, runs evaluators (including cargo check and a quick cargo test), then tears the subprocess down. Per-case timeout is 60s so a hung live model can't stall CI forever.