Expand description
“I fixed the bug”, made checkable.
This is the canonical repair verifier: try, run the tests, retry on
failure, bounded by k. Without it a self-correction loop has nothing to
correct against, and the model is grading its own work.
§Retrying does not launder a failure into a pass
Every other test runner treats a retry as a second chance: run it again, and if it passes this time, it passed. That is the right default for CI, where the cost of a false failure is a re-run. It is the wrong default here, because this verdict gates a commit, and a suite that failed once and passed once has not shown the change to be correct. It has shown the suite to be unreliable, which is a different fact and a more important one.
So retries exist to detect flakiness, not to survive it:
| What happened | Verdict |
|---|---|
| the first run passed | Pass |
| every attempt failed | Fail, with the failing test names |
| attempts disagreed | Inconclusive — the suite is flaky |
| any attempt timed out | Inconclusive |
| the sandbox refused, or the runner would not start | Inconclusive |
With the default of no retries the third row cannot happen and the checker behaves exactly as expected. Turning retries on buys the ability to tell “broken” from “flaky”, and pays for it by refusing to commit on the latter. That is the trade the whole project makes everywhere else: what cannot be proved is not assumed.
§Failing test names, then the tail of the output
A repair loop needs to know which tests failed before it needs to know
what they printed, so the names are extracted first and lead the reason.
failing_tests recognises the formats cargo test, pytest, go test,
and the jest-family runners emit.
What follows is the tail of the output, not the head — the opposite of the compile checker. A test runner puts its summary at the end; a compiler puts its cause at the start. Getting this backwards produces a reason full of passing tests with the failures cut off.
Structs§
- Sandboxed
Tests - Runs a project’s tests and reports whether they passed.
- Test
Config - A test command to run.
Constants§
- MAX_
NAMES - How many failing test names are listed before the rest are counted.
- MAX_
REASON - How much of a test runner’s output reaches the failure reason.
Functions§
- failing_
tests - Extract the names of failing tests from a runner’s output.
- tail
- Keep the tail of
text, which is where a test runner’s summary is.