Module verify

Expand description

The judge’s vocabulary: check results, baseline comparison, and verdicts .

Pure: running checks is the runtime’s job; deciding what their results mean happens here, where it is property-tested.

Structs§

CheckResult: One check run.
Comparison: What changed relative to a baseline (pass-to-pass).
Regression: A test that passed in the baseline and no longer does.
RunnerSummary: Counts read from a test runner’s own summary output (libtest, pytest, jest/vitest, go test), when it writes no JUnit report.
TestResult: One test.
VerifyReport: Everything one verify run established.

CheckStatus: Outcome of one check.
HeldOut: What the held-out acceptance tests said about an attempt that passed visibly: the judge’s own tests, never shown to the model.
RedClass: What a red run of acceptance tests showed.
TestOutcome: Outcome of one test from a JUnit report.
Verdict: How strong the evidence for a task is.

classify_red: Classifies a red run: a JUnit report first, then the runner’s summary, then the check’s status. Nothing recognised is RedClass::Unclassified.
compare: Compares current with baseline, test by test and check by check.
passes: Whether a run passes: no unexplained failing check and no regression.
verdict: The verdict for a task whose task-specific checks are named in task_checks.
with_held_out: The verdict after the held-out tests: a failure fails it (naming the scenarios only); anything else leaves it as it was.