newt-eval 0.8.0

End-to-end evaluation framework for the newt worker
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

//! `newt-eval` — end-to-end evaluation framework for the `newt worker` binary.
//!
//! This crate is the **scorecard** for the worker. It answers:
//! "does this version of the worker actually produce useful patches?"
//!
//! It is also the dogfood substrate for the drake-arbiter scoring pattern —
//! the same shape of test-case + evaluators + scorecard that the arbiter
//! uses to grade swarm candidates.
//!
//! # Two modes
//!
//! - **Mock mode** — runs in CI. A `wiremock` HTTP server stands in for
//!   Ollama and returns a canned diff. The runner spawns the real `newt
//!   worker` binary, drives it over ACP stdio JSON-RPC, and the evaluators
//!   verify the worker did the right thing with the canned response. Fully
//!   deterministic.
//! - **Live mode** — opt-in via `just eval`. No mock — the worker talks to
//!   a real local Ollama. Used to compare models / track regressions on a
//!   developer workstation. **Not** a CI gate.
//!
//! # Shape
//!
//! - [`TestCase`] — loaded from `cases/NNN-name/case.toml` plus a
//!   `workspace/` fixture directory.
//! - [`Evaluator`] — trait; one instance per check (diff non-empty, diff
//!   applies, rust compiles, tests pass, regex pattern match).
//! - [`EvalContext`] — what an evaluator sees after the worker runs.
//! - [`EvalResult`] / [`Scorecard`] — the output rows + their roll-up.
//! - [`runner`] — spawns the `newt worker` subprocess and drives ACP.
//!
//! See `newt-eval/README.md` for how to add a new case.

pub mod cases;
pub mod evaluators;
pub mod grade;
pub mod lint;
pub mod normalize;
pub mod runner;
pub mod score;
pub mod scorecard;
pub mod worker_bin;

#[cfg(feature = "pyo3")]
pub mod pyo3_module;

pub use cases::{default_cases_dir, load_all, MockResponse, TestCase};
pub use evaluators::{
    default_evaluators, evaluator_by_name, CommandRunner, DiffAppliesEvaluator,
    DiffNonemptyEvaluator, Evaluator, OutputMatchesEvaluator, PatternMatchEvaluator, RunSpec,
    RustCompilesEvaluator, SubprocessRunner, TestsPassEvaluator,
};
pub use grade::{
    grade_behavioral, grade_behavioral_with, grade_workspace, harness_subversion, pre_run,
    run_spec, spec_env, verdict_from_run, BehavioralGrade, BehavioralVerdict, PreRun,
    GRADE_SPEC_TIMEOUT_MS,
};
pub use lint::{lint_case_prompts, lint_prompt, LintError};
pub use runner::{run_case, RunOutcome, RunnerConfig};
pub use scorecard::{CaseScorecard, EvalContext, EvalResult, Scorecard};
pub use worker_bin::{resolve_worker_bin, resolve_worker_bin_with, Resolution, ResolutionSource};