# Developer Parity
The question is not "is Procyon better than the tool next to it". It is **how far a developer gets
on a real Stellar task, and how much of that distance the harness is responsible for.** A number
that only compares harnesses tells you which one won a benchmark; this is trying to tell you whether
someone who has never seen Soroban can ship.
```bash
procyon --bench # the whole suite; report on stdout, progress on stderr
procyon --bench build-a-counter # one task, while writing it
procyon --bench > report.toml # commit it, diff the next one against it
```
Exit status is zero only if every task was accepted.
## What a task is
A directory under `bench/tasks/` with:
| `task.toml` | `name`, `prompt`, optional `timeout_secs`, optional `verify`/`seed` overrides |
| `verify.sh` | **the** acceptance criterion — exits zero or the task failed |
| `workspace/` | optional seed, copied into a throwaway directory per run. Absent means the run starts from an empty directory, which is the newcomer's actual starting point |
`prompt` is what the developer would have typed, not an instruction sheet for the model. A prompt
that names the tools to call measures whether the harness can follow orders, which is not the thing
in question.
**Acceptance is a script and nothing else.** Not a rubric, not the model's own account of its work,
not a judge model. A harness that grades itself is the failure this design exists to avoid, so
`verify.sh` checks the artifacts: does the wasm exist, does `cargo test` pass, does the contract id
in the artifacts resolve. A task whose acceptance script is missing is refused at load rather than
counted as a pass — a suite of unfailable tasks reports a perfect score for doing nothing.
## What gets measured
Machine-settled, per task:
- **accepted** — the acceptance script's verdict. The only thing that decides pass or fail.
- **seconds** — wall clock, the closest honest stand-in for time-to-MVP on one task.
- **tool_calls** / **failed_calls** — what the attempt cost, and how much of it was wasted.
- **unsupported_calls** — calls naming a tool the harness does not have. This is the only part of
"hallucination rate" that is not a judgement call, and it is reported under its own name rather
than dressed up as the whole thing.
- **error** — the attempt could not be scored: a timeout, a provider refusal. Deliberately distinct
from `accepted = false`, which means it ran and the work was not good enough.
Not measured here, on purpose: final code quality, whether the workflow chosen was the right one,
and hallucinations about how Stellar behaves. Those need a reviewer. This prints the evidence for
one instead of pretending to be one.
## The arms, and the point of them
A single column of numbers says nothing. The comparison the spec asks for is:
| Stellar newcomer | | | |
| generalist developer | | | |
| Stellar expert / DevRel | | | |
The expert row is the one that matters most and is the easiest to skip: **parity is the target, not
victory.** If a newcomer running Procyon lands near where the expert lands unaided, the harness did
its job. If Procyon beats the expert, the task was too small to be telling.
Run the arms with the same tasks and the same seeds; only the harness changes. `procyon --knowledge`
snapshots the skills and endpoints a run was working from, so a difference between two runs can be
traced to a difference in inputs rather than blamed on the model.
## Fixed by construction
Every run happens in a throwaway copy of `workspace/`, so a task cannot be passed by artifacts
another task left behind, and a failed run cannot corrupt the seed.
Runs pass `--allow-changes`: a task nobody may write for cannot be completed. That does not reach
the refusals — mainnet signing and secret keys in arguments are `Deny`, not `Confirm`, and a
benchmark able to sign on mainnet is a benchmark nobody should run.