Module eval 

speccoder eval: judge attempts with held-out oracles.

A case is case.toml + project/ + oracle/. Each runs in a fresh git repository and state directory; the oracle is copied into the attempt’s worktree only after the attempt has ended, so the agent never sees it.

Constants

SUITE

The suite configuration file.

Functions

cases

The cases of a suite, in name order.

run_suite

Runs the suite at dir (optionally one case) repeat times with provider. Repeated rounds keep their cases under keep/round-<n>/.

run_suite_as

run_suite with every case’s attempts made by an external executor (an ACP agent’s preset id), to compare it with the native loop on the same cases and oracles.

suite_config

The suite’s [model] and [budget] text, appended to each case’s config.