Skip to main content

RUNBOOK_TEMPLATE

Constant RUNBOOK_TEMPLATE 

Source
pub const RUNBOOK_TEMPLATE: &str = "# Eval run \u{2014} {{SKILL_NAME}} (iteration {{ITERATION}}, {{HARNESS}})\n\nThis runbook is for a human driving the run from a terminal. Work from this iteration directory\nand copy-paste each step. The workspace is self-contained \u{2014} you should not need the surrounding\nrepo.\n\n- **Skill under test:** {{SKILL_NAME}}\n- **Mode:** {{MODE}} \u{2014} comparing `{{COND_A}}` vs `{{COND_B}}`\n- **Dispatches:** {{NUM_TASKS}} (the `tasks[]` array in `{{DISPATCH_JSON}}`)\n\n## 1. Dispatch the eval agents, then ingest\n{{DISPATCH_RECIPE}}\n\n`ingest` records each run, backfills transcripts, scans for stray writes, and grades every\nmechanical assertion. It then prints any `llm_judge` tasks it could not grade itself.\n\n## 2. Dispatch the judge agents, then finalize\n{{JUDGE_RECIPE}}\n\nThen merge the verdicts and aggregate:\n\n```\n{{FINALIZE_CMD}}\n```\n\n## 3. Read the result\n\n`finalize` writes the cross-condition benchmark to:\n\n```\n{{BENCHMARK_PATH}}\n```\n\nRead it for the per-condition pass rates and the `{{COND_A}}` \u{2212} `{{COND_B}}` deltas.\n\n## 4. Tear down\n\n```\n{{TEARDOWN_CMD}}\n```\n";
Expand description

The shared (human-followed) RUNBOOK.md template used by every run, regardless of harness (Claude Code, Codex, OpenCode).