# newt-eval — end-to-end evaluation framework for the newt worker
`newt-eval` is the scorecard for `newt worker`. It answers:
> Does this version of the worker actually produce useful patches?
It is also the dogfood substrate for the drake-arbiter scoring pattern
— the same test-case + evaluators + scorecard shape the arbiter uses
to grade swarm candidates.
## Two modes
| Mock | `cargo test -p newt-eval --test mock_e2e` | `wiremock` stand-in for Ollama; the real `newt` binary drives ACP | yes — runs in `just check` |
| Live | `just eval` (alias for `newt-eval run --mode live`) | A real local Ollama via `OLLAMA_HOST` or the worker's default endpoint list | no — opt-in developer tool |
Mock mode is fully deterministic — every case ships with a canned diff
that the wiremock returns verbatim. The evaluators then verify the
worker did the right thing with that response (parsed it, applied it
to the workspace, captured the resulting git diff, returned a
well-formed `TaskReply`).
Live mode is for comparing models / catching regressions: the worker
runs against your real Ollama and the same evaluators grade whatever
the model returns. Weak models will legitimately fail — that's the
point.
## Running
```bash
# Mock e2e (runs in CI):
cargo test -p newt-eval --test mock_e2e
# List the bundled cases:
cargo run -q --bin newt-eval -- list-cases
# Live: all cases against discovered Ollama:
just eval
# Live: one case, specific model:
just eval --case 001 --model llama3.1:8b
# Coder mode + raised timeout for a slower local model:
newt-eval run --mode live --case 008 --coder --worker-timeout-ms 180000
```
### Coder mode and slow models
- `--coder` spawns the worker with the `newt-coder` plugin (whole-file
emit + server-side diff normalization). Use it for local Ollama coder
models that otherwise trip failure mode T0b (invented hunk context).
- `--worker-timeout-ms <ms>` (env `NEWT_EVAL_WORKER_TIMEOUT_MS`, default
`60000`) is the per-case wall-clock budget. A model slower than this is
scored `dispatch_error` even if it would have produced correct output —
raise it (e.g. `180000`) when evaluating slower models so the scorecard
measures capability, not just latency.
Exit codes from `newt-eval run`:
- `0` — every case passed every evaluator
- `2` — ran cleanly but at least one case failed
- `1` — hard error (worker not found, cases dir missing, etc.)
## The five evaluators
| `diff_nonempty` | `reply.diff` is non-empty AND `!reply.empty_diff` |
| `diff_applies` | Copy baseline to a tempdir, `git apply --check` accepts the model's **raw emission** (when diff-shaped, fence peeled) — not the post-hoc captured diff — so a header-lying diff the fuzzy worker rescued is scored honestly (#30B) |
| `rust_compiles` | `cargo check` on the post-worker workspace (Rust cases only) |
| `tests_pass` | `cargo test` on the post-worker workspace (Rust cases with `#[test]` only) |
| `pattern_match` | At least one of `expected_patterns` regex matches the captured diff |
Language-specific evaluators auto-skip non-Rust cases with a clear note.
`tests_pass` also skips Rust cases that have no `#[test]` anywhere so
trivial cases don't pay the cargo-test bill.
## The `output_matches` result oracle (#957)
Beyond the diff/pattern checks, `output_matches` grades **what the program
prints** — it runs the graded program and compares its stdout to a declared
`expected_output`. It is **opt-in** (add `"output_matches"` to a case's
`evaluators`; it is never in the defaults) and passes with a skip note when a
case declares no `expected_output` / `[output_match]`.
A case configures it with a top-level `expected_output` and an `[output_match]`
table:
```toml
evaluators = ["output_matches"]
# What the program's stdout must equal (after the normalize pipeline below).
expected_output = "42"
[output_match]
# argv to run in the graded worktree. argv[0] is the program/interpreter.
run = ["python3", "solution.py"]
# Hard wall-clock budget; the process is killed on expiry and reported timed_out.
timeout_ms = 10000
# Ordered normalization strategies (#959), applied to BOTH sides before compare.
# Empty list = exact byte match.
normalize = ["regex_extract", "numeric_tolerance"]
# For regex_extract: reduce each side to capture group 1 (pull the answer out of
# surrounding noise). Program may print "computing...\nResult: 42.0\n".
extract_pattern = 'Result:\s*([\d.]+)'
# For numeric_tolerance: compare as numbers within epsilon (default 1e-9).
epsilon = 0.001
```
**Normalization strategies** (pure data — new tolerance is config, not code):
| `trim` | strip leading/trailing whitespace |
| `collapse_whitespace` | runs of whitespace → a single space |
| `trailing_newline` | drop a trailing CR/LF run |
| `regex_extract` | reduce each side to capture group 1 of `extract_pattern` |
| `numeric_tolerance` | compare as `f64` within `epsilon`; falls back to string compare when non-numeric |
Unknown strategy names, and `regex_extract` with a missing/invalid
`extract_pattern`, are **skipped with a warning** (never fatal) — the same
tolerant-load rule the language packs use.
**Trust boundary + tiering.** The real runner (`SubprocessRunner`) executes
untrusted, model-produced code and does **not** sandbox it (no network or
filesystem confinement beyond `cwd`; see #887). The real-subprocess tests live
in [`tests/output_matches_real.rs`](tests/output_matches_real.rs), are
`#[ignore]`d out of the per-PR unit run, and run in the **weekly + release**
tiers, single-threaded:
```bash
cargo test -p newt-eval --test output_matches_real -- --ignored --test-threads=1
```
(CI: `.github/workflows/output-oracle-real.yml`.)
## Adding a new case
A case is a directory under `newt-eval/cases/NNN-name/` with two parts:
1. `case.toml` — metadata + mock response.
2. `workspace/` — the initial filesystem state, copied verbatim into a
tempdir at the start of each run.
> See [`cases/CASE_AUTHORING.md`](cases/CASE_AUTHORING.md) for the full
> guide — the emission-contract rule and how to generate byte-accurate
> goldens from `git diff`.
```toml
# newt-eval/cases/010-your-case/case.toml
name = "010-your-case"
description = "One-line summary"
language = "rust"
difficulty = "L1" # L1 saturated edit · L2 multi-step · L3 cross-domain
prompt = """
Multi-line instruction describing the TASK only. Do NOT specify the
response format — the system prompt owns the emission shape, and a
`case_prompt_lint` test fails on directives like "respond with a
unified diff only" (see CASE_AUTHORING.md).
"""
evaluators = [
"diff_nonempty",
"diff_applies",
"rust_compiles",
"tests_pass",
"pattern_match",
]
# At least one of these regexes must match the captured diff.
expected_patterns = [
"fn new_thing\\(",
]
# Mock-mode response — exactly what the wiremock will return as the
# Ollama assistant content. Must be a valid unified diff against the
# workspace/ fixture so the worker's apply_patch can take it.
[mock_response]
content = """
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,3 +1,3 @@
-old line
+new line
"""
```
Verify a new case before committing:
```bash
# 1. The case loads cleanly:
cargo run -q --bin newt-eval -- list-cases
# 2. The mock diff would actually apply to the fixture
# (independent of the worker — sanity check):
cp -r newt-eval/cases/006-your-case/workspace /tmp/v
cd /tmp/v && git init -q && git add -A && git commit -q -m b
git apply --check /path/to/your.diff && echo OK
# 3. End-to-end with the worker:
cargo test -p newt-eval --test mock_e2e
```
## Test budget
The mock e2e test runs all bundled cases in well under 30 seconds in CI
once the worker binary is built. Each case spawns a worker subprocess,
drives ACP, runs evaluators (including `cargo check` and a quick
`cargo test`), then tears the subprocess down. Per-case timeout is 60s
so a hung live model can't stall CI forever.