Expand description
Evals by replay: re-run recorded runs on a candidate configuration (e.g. another model), score the results and compare cost.
Replays never repeat real side effects. A tool call that matches one in
the recorded run gets the recorded result. An unmatched Read call may
run live (LiveTools::ReadOnly, the default); unmatched Write and
Irreversible calls are never executed and the model is told so.
Replays use their own in-memory journal, never the production one.
Structs§
- Case
Result - Completed
- 1 if the candidate run completed (including any finish schema).
- Eval
Case - One recorded run to replay.
- Eval
Report - Evaluator
- Replays cases against a candidate configuration derived from an agent.
- Fields
Match - Fraction of these structured-output fields equal to the baseline’s.
- Judge
- An LLM grades the candidate’s answer against the baseline’s.
- RunSummary
- What a run achieved: the recorded baseline or a replayed candidate.
- Score
- A 0–1 score with an explanation.
Enums§
- Live
Tools - Whether unmatched tool calls may execute during a replay.