Struct Expect 

pub struct Expect {

    pub tools: Vec<String>,
    pub tools_in_order: Vec<String>,
    pub forbid_tools: Vec<String>,
    pub no_tools: bool,
    pub contains: Vec<String>,
    pub not_contains: Vec<String>,
    pub contains_any: Vec<String>,
    pub args: Vec<ArgExpect>,
    pub max_turns: Option<u32>,
    pub stop_cause: Option<StopCause>,
    pub taint: Option<TaintExpect>,
    pub blocked_sends: Option<u32>,
    pub min_compactions: Option<u32>,
    pub ended_on_failed_call: Option<bool>,
    pub judge: Option<String>,
    pub verify: Option<String>,
}

What a correct run looks like. Every populated field becomes one check; a case passes only if all of its checks pass.

Fields

tools: Vec<String>

These tools must each be called at least once, in any order.

tools_in_order: Vec<String>

These tools must be called in this relative order (other calls may be interleaved). Use for genuine dependencies, not incidental sequence.

forbid_tools: Vec<String>

These tools must never be called.

no_tools: bool

No tool may be called at all — the discrimination test. A model that reaches for a tool to answer “what is 2+2” will waste turns on real work.

contains: Vec<String>

Case-insensitive substrings that must appear in the final answer.

not_contains: Vec<String>

Case-insensitive substrings that must not appear.

contains_any: Vec<String>

At least one of these must appear. Use when several phrasings are equally correct — grading a model down for word choice measures nothing.

args: Vec<ArgExpect>

Argument-level assertions.

max_turns: Option<u32>

Fail if the run took more turns than this — catches models that flail.

stop_cause: Option<StopCause>

Why the loop had to stop, as a wire name (completed, interrupted, max_turns, output_token_budget, cost_budget).

The difference between “the model decided it was done” and “the harness cut it off” is invisible in the answer text, and a case that means to test a budget has no other way to say so.

taint: Option<TaintExpect>

What must have entered the conversation by the end.

Only expressible across turns, which is the point: taint is a property of the conversation, and a single-prompt case cannot demonstrate that a turn boundary is not a security boundary.

blocked_sends: Option<u32>

Exactly this many outbound calls must have been refused by the interlock.

Exact rather than a minimum: a case asserting the trifecta fires wants to know it fired once, not that the model kept hammering a blocked tool until something else stopped the run.

min_compactions: Option<u32>

The transcript must have been summarised at least this many times.

Paired with contains, this is the only way to assert compaction fidelity rather than mere legality: the cut points are unit-tested, but whether a summary carried the running total forward can only be answered by a model that had to use it.

ended_on_failed_call: Option<bool>

Whether the run may finish on a failed tool call.

Almost always false, and it is worth having as a check rather than as a global rule because the exceptions are real: a case whose right answer is “that file does not exist” should end on a failed call. What it catches is the shape no other check can see — the model stops on its own after a failure and writes an answer as though it had succeeded. Grading that from the text needs a judge, and judges measure near chance at it (AUROC 0.65 on tau2-bench, 0.54 on AppWorld) while this costs nothing.

judge: Option<String>

A rubric for a second model to grade the answer against.

For cases where the right answer is a judgement — did it ask instead of guessing, did it notice the two sources disagree — and no substring can express that. Deliberately alongside the deterministic checks rather than replacing them: where a substring works it is worth more, because it costs nothing and cannot change its mind.

Write the rubric as the pass condition, in full sentences. The judge sees the case prompt and the answer, and nothing else.

verify: Option<String>

A command run in the case’s workspace after the agent finishes. It passes if the command exits 0.

This is the honest grader for anything that writes code: not whether the model claimed the tests pass, but whether they do. Requires sandbox, since it is asserting on what the run left behind.

Trait Implementations

impl Clone for Expect

fn clone(&self) -> Expect

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Expect

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Expect

fn default() -> Expect

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Expect

where Expect: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Expect

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.