pub struct OutcomeEvalset {
pub hash: String,
pub field: String,
pub higher_is_better: bool,
pub horizons_ms: Vec<i64>,
pub checkpoints: Vec<Checkpoint>,
pub baseline: BaselineKind,
pub min_effect: Option<MinEffect>,
pub cost: Option<CostBound>,
}Expand description
The evalset every LLM-authored, applicable proposal is measured against
after apply (docs/loop.md, “Evalset-backed outcomes”). An authored
lesson carries no built-in recurrence metric — nothing errors when a
lesson is merely useless — so without this the Verify gate has nothing
to re-measure for exactly the proposals a human was least able to judge.
The host names the evalset and the field; the engine takes the baseline
from the newest run journaled BEFORE the proposal and reads the current
value from runs journaled AFTER the apply. No baseline run → no metric,
never a fabricated one.
Fields§
§hash: StringThe evalset hash (the subject is evalset:<hash> in agent:harness).
field: StringThe summary field to read: passed, failed, total, error_rate,
or any numeric field the host’s harness writes into the summary.
higher_is_better: boolWhich direction is an improvement — passed and an accuracy are
higher-is-better, failed and error_rate are not. Stated by the
host because getting it wrong would revert an improvement.
horizons_ms: Vec<i64>Checkpoints after apply, in ms. Default 1d / 7d / 30d. The older
spelling; checkpoints wins when both are given.
checkpoints: Vec<Checkpoint>The schedule in the deployment’s own unit — {"after_ms": n},
{"after_runs": n} or {"after_grains": n} (a bare integer is ms).
A benchmark or CI harness wants [{"after_runs": 1}]: measure at the
next graded run after the apply, however soon that is. Empty (the
default) means horizons_ms.
baseline: BaselineKindWhich run before the apply the verdict compares against (default
newest_before_apply; see BaselineKind for why high_water is
opt-in).
min_effect: Option<MinEffect>The minimum effect size a verdict needs to call a regression (default none: any drop past floating-point slack is a regression).
cost: Option<CostBound>A cost bound read beside the quality field (default none).
Implementations§
Source§impl OutcomeEvalset
impl OutcomeEvalset
Sourcepub fn schedule(&self) -> Vec<Checkpoint>
pub fn schedule(&self) -> Vec<Checkpoint>
The effective schedule: checkpoints when set, else horizons_ms as
time checkpoints.