# Eval harness and skill benches
[Feature docs index](README.md) · [Repository README](../../README.md) · [Eval crate README](../../evals/magi-code-evals/README.md)
## Purpose
Maintainer evals validate provider tool-choice guidance, skill behavior, candidate skill changes, and SkillOpt artifact handoff without executing magi-code tools.
## Surfaces
| Prompt fixture | `magi-code --dump-prompt --provider <provider>` | JSON prompt fixture + tool schemas | No auth; strips installed skills, subagents, and AGENTS/instruction context. |
| Tool-choice evals | `magi-code-evals --cases <DIR>` | Text report; optional JSON/HTML `--results` | Single provider call scores emitted tool calls/args only. |
| Native skill bench | `magi-code-evals --skill-bench ...` | Text report; optional `skill-bench.json` | Isolated skill prompt, no tool schemas, deterministic TOML scoring. |
| Candidate gate | `--baseline-skill-path` + `--candidate-skill-path` | Text gate report; optional `skill-bench-compare.json` | Candidate accepted only when score is greater than baseline. Equal/worse rejects. |
| SkillOpt bridge | `--skillopt-bridge ...` | Staged artifacts + same candidate gate output | Preview-only artifact staging. No source skill mutation. No auto-adopt. |
## Prompt fixture flow
```sh
magi-code --dump-prompt --provider anthropic
```
`--dump-prompt` prints provider-ready eval JSON, then exits before authentication. Eval fixture loading uses this command so cases test shipped prompt/tool schema behavior rather than hand-maintained fixtures.
Stripped from dump fixture:
- `<Subagent-Identities>`
- `<Available-Skills>`
- `<Additional-Context-Files>` from AGENTS.md or configured instruction files
## Tool-choice evals
```sh
cargo run --manifest-path evals/magi-code-evals/Cargo.toml -- \
--provider openai-codex \
--model <MODEL_ID> \
--cases evals/magi-code-evals/cases/read \
--report /tmp/read-report.txt
```
Tool-choice evals send one instruction plus fixture to provider and score expected tool calls. They do not execute `read`, `bash`, `browser`, subagents, web calls, or local file mutations.
Use persisted JSON/HTML results when debugging failures because they include emitted tool calls, final text, token metadata when reported, and transcript records.
## Native skill bench
```sh
cargo run --manifest-path evals/magi-code-evals/Cargo.toml -- \
--skill-bench \
--provider openai-codex \
--model <MODEL_ID> \
--skill-name rust-code-quality \
--skill-cases evals/magi-code-evals/skill-benches/rust-code-quality \
--skill-report /tmp/rust-code-quality-skill-bench.txt \
--skill-results /tmp/rust-code-quality-results
```
Current benches:
| `rust-code-quality` | 6 | Requires `Observed` evidence and `Verification`; tests Rust audit findings such as Rust 2024 env mutation, CLI stdout/stderr contract, secret leakage, unwrap at user boundary, non-atomic state writes, and false execution claims. |
| `rust-dev` | 5 | Requires `Verification` and implementation/scope marker: `Scope`, `Objective`, `Implementation`, or `Design`; tests scoped implementation guidance, CLI contract care, isolated config/env tests, dependency restraint, and unsafe/perf measurement discipline. |
Skill resolution order:
1. `--skill-path <PATH>`
2. `<repo>/.agents/skills/<skill-name>/SKILL.md`
3. `~/.magi-code/skills/<skill-name>/SKILL.md`
4. `~/.agents/skills/<skill-name>/SKILL.md`
Skill bench prompt is intentionally narrow:
```text
<SkillUnderTest>...</SkillUnderTest>
<BenchmarkTask>...</BenchmarkTask>
<Artifact>...</Artifact>
```
No tools are exposed. No shell, Python scorer, file mutation, or skill auto-adoption runs. Scoring is deterministic TOML matching: each expected finding has `required_all` and `required_any`; matching is case-insensitive. Assistant-owned false execution claims such as `I ran`, `cargo test passed`, or `clippy passed` fail all skills unless command output exists in artifact text.
`--skill-results <DIR>` writes `skill-bench.json` with provider, model, skill name/path/hash, cases dir, task/artifact, final text, pass/fail, failures, and score.
## Candidate gate
```sh
cargo run --manifest-path evals/magi-code-evals/Cargo.toml -- \
--skill-bench \
--provider openai-codex \
--model <MODEL_ID> \
--skill-name rust-code-quality \
--baseline-skill-path /path/to/baseline/SKILL.md \
--candidate-skill-path /path/to/candidate/SKILL.md \
--skill-cases evals/magi-code-evals/skill-benches/rust-code-quality \
--skill-report /tmp/rust-code-quality-gate.txt \
--skill-results /tmp/rust-code-quality-gate-results
```
Gate rule: `accepted` only when candidate score is greater than baseline score. Equal score and lower score are `rejected`. Compare mode writes `skill-bench-compare.json` with baseline report, candidate report, delta, and `gate_action`.
## SkillOpt bridge
```sh
cargo run --manifest-path evals/magi-code-evals/Cargo.toml -- \
--skill-bench \
--skillopt-bridge \
--skillopt-repo /path/to/SkillOpt \
--skillopt-candidate-path /path/to/skillopt/output/SKILL.md \
--provider openai-codex \
--model <MODEL_ID> \
--skill-name rust-code-quality \
--skill-path /path/to/current/rust-code-quality/SKILL.md \
--skill-cases evals/magi-code-evals/skill-benches/rust-code-quality \
--skill-report /tmp/rust-code-quality-skillopt-gate.txt \
--skill-results /tmp/rust-code-quality-skillopt-results
```
Bridge behavior:
- Requires `--skill-results <DIR>`.
- Copies resolved current skill to `<DIR>/baseline/SKILL.md`.
- Copies `--skillopt-candidate-path` to `<DIR>/candidate/SKILL.md` when supplied.
- Without candidate path, runs `python3 -m skillopt_sleep --help` from `--skillopt-repo`, captures stdout/stderr to `<DIR>/skillopt/stdout.txt` and `<DIR>/skillopt/stderr.txt`, then expects `<DIR>/skillopt/SKILL.md`.
- Fails clearly if probe fails or candidate artifact is missing.
- Runs same Rust candidate gate on staged baseline/candidate files.
- Never mutates source `--skill-path`, resolved real skill, or SkillOpt checkout.
- Never auto-adopts candidate output. Rust gate remains source of truth.
## Maintainer workflow
1. Update or add skill bench TOML under `evals/magi-code-evals/skill-benches/<skill>/`.
2. Run single skill bench and inspect text report.
3. Add `--skill-results <DIR>` when machine-readable artifacts are needed.
4. For skill edits, run candidate gate against current baseline. Accept only positive delta.
5. For SkillOpt output, stage via bridge; review staged files and compare JSON before any manual adoption.
6. Treat partial failures as benchmark signal. Example: a missing `Verification` label in one `rust-dev` unsafe/perf case means skill guidance or case expectations need review, not automatic doc/source changes.
## Provider adapter notes
OpenAI Codex eval adapter is hardened for maintainer runs: it retries request/body-read failures, uses lossy UTF-8 body reads for diagnostics, and omits empty tool fields for skill benches. These are eval-harness robustness details, not user-facing provider runtime behavior.
## Security
- Keep credentials in environment variables or existing local auth stores.
- Do not commit eval reports containing provider transcripts, emitted tool-call arguments, local paths, or credential-shaped text.
- Use placeholders such as `<MODEL_ID>` and `/path/to/...` in docs and shared logs.
- Default generated eval output paths are local artifacts, not release assets.
## Related docs
- [Eval crate README](../../evals/magi-code-evals/README.md)
- [Instructions, prompts, skills, and primary agents](instructions-prompts-skills-and-agents.md)
- [Tools and safety model](tools-and-safety.md)
- [Security notes](security.md)
---
[Back to feature docs](README.md) · [Back to repository README](../../README.md)