Module verify 

The judge’s vocabulary: check results, baseline comparison, and verdicts .

Pure: running checks is the runtime’s job; deciding what their results mean happens here, where it is property-tested.

Structs

CheckResult

One check run.

Comparison

What changed relative to a baseline (pass-to-pass).

Regression

A test that passed in the baseline and no longer does.

RunnerSummary

Counts read from a test runner’s own summary output (libtest, pytest, jest/vitest, go test), when it writes no JUnit report.

TestResult

One test.

VerifyReport

Everything one verify run established.

Enums

CheckStatus

Outcome of one check.

HeldOut

What the held-out acceptance tests said about an attempt that passed visibly: the judge’s own tests, never shown to the model.

RedClass

What a red run of acceptance tests showed.

TestOutcome

Outcome of one test from a JUnit report.

Verdict

How strong the evidence for a task is.

Functions

classify_red

Classifies a red run: a JUnit report first, then the runner’s summary, then the check’s status. Nothing recognised is RedClass::Unclassified.

classify_red_added

Classifies a red run against the same command run without the file under test: red by an assertion only where the file adds a failing test — one that did not fail without it (by id when both runs name their tests, else by more failures). A failure the command shows anyway is another test’s, never this file’s red: it is RedClass::Passed.

classify_red_each

The class of each test in wanted from a run’s JUnit report: a test matches by its id or its last :: segments; one the report does not name was not collected. None when the run has no per-test report — then each test is run on its own, by name.

classify_red_one

The class of the one test test from a run of the command that is to run only it: by name from a per-test report; without one, from the runner’s summary only when it ran exactly one test — a run of several tests cannot say which of them failed, so it is RedClass::Unclassified, never an assertion of test’s.

compare

Compares current with baseline, test by test and check by check.

passes

Whether a run passes: no unexplained failing check and no regression.

verdict

The verdict for a task whose task-specific checks are named in task_checks.

with_held_out

The verdict after the held-out tests: a failure fails it (naming the scenarios only); anything else leaves it as it was.