pub struct Expect {Show 15 fields
pub tools: Vec<String>,
pub tools_in_order: Vec<String>,
pub forbid_tools: Vec<String>,
pub no_tools: bool,
pub contains: Vec<String>,
pub not_contains: Vec<String>,
pub contains_any: Vec<String>,
pub args: Vec<ArgExpect>,
pub max_turns: Option<u32>,
pub stop_cause: Option<StopCause>,
pub taint: Option<TaintExpect>,
pub blocked_sends: Option<u32>,
pub min_compactions: Option<u32>,
pub judge: Option<String>,
pub verify: Option<String>,
}Expand description
What a correct run looks like. Every populated field becomes one check; a case passes only if all of its checks pass.
Fields§
§tools: Vec<String>These tools must each be called at least once, in any order.
tools_in_order: Vec<String>These tools must be called in this relative order (other calls may be interleaved). Use for genuine dependencies, not incidental sequence.
forbid_tools: Vec<String>These tools must never be called.
no_tools: boolNo tool may be called at all — the discrimination test. A model that reaches for a tool to answer “what is 2+2” will waste turns on real work.
contains: Vec<String>Case-insensitive substrings that must appear in the final answer.
not_contains: Vec<String>Case-insensitive substrings that must not appear.
contains_any: Vec<String>At least one of these must appear. Use when several phrasings are equally correct — grading a model down for word choice measures nothing.
args: Vec<ArgExpect>Argument-level assertions.
max_turns: Option<u32>Fail if the run took more turns than this — catches models that flail.
stop_cause: Option<StopCause>Why the loop had to stop, as a wire name (completed, interrupted,
max_turns, output_token_budget, cost_budget).
The difference between “the model decided it was done” and “the harness cut it off” is invisible in the answer text, and a case that means to test a budget has no other way to say so.
taint: Option<TaintExpect>What must have entered the conversation by the end.
Only expressible across turns, which is the point: taint is a property of the conversation, and a single-prompt case cannot demonstrate that a turn boundary is not a security boundary.
blocked_sends: Option<u32>Exactly this many outbound calls must have been refused by the interlock.
Exact rather than a minimum: a case asserting the trifecta fires wants to know it fired once, not that the model kept hammering a blocked tool until something else stopped the run.
min_compactions: Option<u32>The transcript must have been summarised at least this many times.
Paired with contains, this is the only way to assert compaction
fidelity rather than mere legality: the cut points are unit-tested,
but whether a summary carried the running total forward can only be
answered by a model that had to use it.
judge: Option<String>A rubric for a second model to grade the answer against.
For cases where the right answer is a judgement — did it ask instead of guessing, did it notice the two sources disagree — and no substring can express that. Deliberately alongside the deterministic checks rather than replacing them: where a substring works it is worth more, because it costs nothing and cannot change its mind.
Write the rubric as the pass condition, in full sentences. The judge sees the case prompt and the answer, and nothing else.
verify: Option<String>A command run in the case’s workspace after the agent finishes. It passes if the command exits 0.
This is the honest grader for anything that writes code: not whether the
model claimed the tests pass, but whether they do. Requires sandbox,
since it is asserting on what the run left behind.