pub struct RunStats {Show 20 fields
pub turns: u32,
pub usage: Usage,
pub cost_usd: Option<f64>,
pub usage_complete: bool,
pub stop_cause: Option<StopCause>,
pub exhausted: bool,
pub ended_on_failed_call: bool,
pub tool_calls: u32,
pub tool_errors: u32,
pub tool_denied: u32,
pub tool_staged: u32,
pub malformed_tool_args: u32,
pub blocked_sends: u32,
pub compactions: u32,
pub context_overflows: Option<u32>,
pub boredom_notices: Option<u32>,
pub step_escalations_attempted: Option<u32>,
pub step_escalations_revised: Option<u32>,
pub homeostat: Option<Homeostat>,
pub taint: Taint,
}Expand description
How a run went, in numbers a machine can compare across sessions.
Every field is a deterministic count taken from
crate::agent::RunOutcome — nothing here is a model’s opinion, and
nothing is derived from the content of a tool result. That is the
property that lets this be an input to automated grading: a counter
carries no instructions, so a corpus of these cannot be an injection
surface the way a corpus of transcript excerpts would be.
Every field defaults, so a session written before this record existed loads, and a field added later does not invalidate the ones already recorded.
Fields§
§turns: u32§usage: Usage§cost_usd: Option<f64>§usage_complete: boolFalse when usage is a lower bound rather than a measurement.
stop_cause: Option<StopCause>Why the loop stopped. The single most informative field here: it separates “the model decided it was done” from every way the harness cut it short, and none of that is visible in the answer text.
exhausted: bool§ended_on_failed_call: boolThe model stopped of its own accord with its last call failed.
tool_calls: u32Tool calls attempted, and how they went. errors counts the
environment refusing (including a call to a tool that does not exist);
denied counts a human or a policy refusing, which is the harness
working and must not be averaged in with failure.
tool_errors: u32§tool_denied: u32§tool_staged: u32§malformed_tool_args: u32§blocked_sends: u32§compactions: u32§context_overflows: Option<u32>Times a prompt was refused as too large and the run recovered.
Option, unlike every other counter here, and the difference is the
point. This field exists to be a baseline — the thing a change
claiming to predict overflows is measured against — so the measurement
spans the moment it was introduced. A row written before that knows
nothing, and a plain u32 would read it as a run that overflowed zero
times, silently diluting the very rate it was added to establish.
None says the sensor was not there. Absent is not zero, the rule
crate::homeostat and crate::backlog both state at length.
boredom_notices: Option<u32>Times the harness told this run an approach had stopped teaching it
anything (GOAL-SYSTEM-DESIGN.md §9.1).
Option for context_overflows’ reason, one field up: every threshold
behind it was argued rather than measured, and this is the field that
makes them answerable. A row from before the detector existed knows
nothing, and reading it as a run that was never bored would dilute the
rate it was added to establish.
step_escalations_attempted: Option<u32>How many step-escalation candidates (GOAL-SYSTEM-DESIGN.md §5.5)
actually spent a quarantined call this run. boredom_notices’s own
reason: the pre-filter’s thresholds are argued, not measured, and a
row from before the mechanism existed knows nothing.
step_escalations_revised: Option<u32>Of those, how many came back revise_plan.
homeostat: Option<Homeostat>The conditions this run happened under, when the front-end asked for
them. Recorded here rather than derived later because a run
reconstructed against today’s machine state is measuring the
afternoon — see GOAL-SYSTEM-DESIGN.md §12.
taint: TaintWhat had entered the conversation by the end. Recorded here as well as
in Record::Taint because this record is read on its own, by a
reader that is counting rather than reconstructing.
Implementations§
Source§impl RunStats
impl RunStats
Sourcepub fn absorb(&mut self, o: &RunOutcome)
pub fn absorb(&mut self, o: &RunOutcome)
Fold another run’s outcome in.
An episode — a replayed session, a multi-turn eval case, a batch item — is several runs on one conversation, and is one row. Counters sum, because the episode really did spend all of it. Three fields do not, and the split is the whole reason this is a method rather than a loop at each call site:
stop_cause,exhaustedandended_on_failed_calldescribe how the episode ended, so the last run wins. An episode whose first turn ended on a failure and whose second recovered has not finished over a failure, and summing would say it had.taintmerges and never resets: it is a property of the conversation, and a later clean run does not un-read what an earlier one read.usage_completeis an AND: one lower-bound turn makes the total a lower bound.
Sourcepub fn merge(&mut self, other: &RunStats)
pub fn merge(&mut self, other: &RunStats)
Fold another row in, by the rules above.
Same code as absorb, deliberately: an episode is several runs, and
something has to be able to rebuild one from the rows a session
recorded — harness_probe sizes its priority signal that way, and it
has to fold exactly as the arm it will be compared against does. Two
spellings of this is how a measurement arm and the thing it measures
stop being comparable without anyone noticing.
homeostat is untouched, as it always was: the conditions belong to
the run that sampled them, and an episode’s several runs happened under
several. The first one set keeps the field.
Sourcepub fn fold(rows: impl IntoIterator<Item = RunStats>) -> Option<RunStats>
pub fn fold(rows: impl IntoIterator<Item = RunStats>) -> Option<RunStats>
Fold a session’s recorded rows into the episode they describe.
The seed is the first row rather than default(), because
usage_complete is ANDed down — starting from the default’s false
would make every folded episode a lower bound.