Module eval 

Grading agent runs.

Built for one job: deciding which model to run locally. Final text is a poor signal for that — what matters is whether the model picked the right tool, passed well-formed arguments, and stopped when it should have. So cases are graded on the tool-call trace first and the text second.

Cases are read-only against a shared fixture by default. That makes them reproducible, safe to run at high concurrency, and repeatable across models — which is the whole point of a bake-off.

A case that must write — write a function, run the tests, fix what fails — sets sandbox: true and gets a private throwaway copy of the fixture instead. Same reproducibility, because nothing it does is visible to any other case or to the next run.

Structs

ArgExpect

An assertion about the arguments of a particular tool call.

Check

EvalCase

Expect

What a correct run looks like. Every populated field becomes one check; a case passes only if all of its checks pass.

GradedCase

Judge

Grades open-ended answers against a rubric, using a second model.

Scorecard

Aggregate view of one model’s run over the whole case set.

TagScore

TaintExpect

What must have entered the conversation. Unset legs are not asserted on.

Verdict

What the judge decided. reason is recorded in the report so a surprising verdict can be argued with rather than just believed.

Functions

grade

Grade one result against its case.

stage_workspace

Copy a fixture into a private directory for one sandboxed case.

verify_workspace

Run a case’s verify command in its workspace and grade the exit code.