Struct EvalCase 

pub struct EvalCase {
    pub id: String,
    pub prompt: Prompt,
    pub expect: Expect,
    pub tags: Vec<String>,
    pub sandbox: bool,
    pub max_turns: Option<u32>,
    pub compact_at_tokens: Option<u64>,
}

Fields

id: String

prompt: Prompt

One turn, or several run on the same conversation.

expect: Expect

tags: Vec<String>

Free-form labels. The scorecard breaks results down by tag, which is how you see where a model falls over rather than just how often.

sandbox: bool

Run this case against a private copy of the fixture, with writing tools allowed.

Off by default and deliberately explicit per case: a case set where anything might mutate the shared fixture is a case set where run N and run N+1 measure different things.

max_turns: Option<u32>

Turns this case may take, when the default budget is not enough.

A case that genuinely needs twenty steps should say so. The alternative — raising the global ceiling for one case — quietly changes what every other case in the set is allowed to do, and max_turns is one of the things being measured.

compact_at_tokens: Option<u64>

Compact this case’s transcript at this many reported prompt tokens.

A compaction case has to force the behaviour it is grading, and it must do so for itself alone: turning compaction on globally would quietly change what every other case in the set is measuring.

Implementations

impl EvalCase

pub fn validate(&self) -> Result<()>

Catch case-file mistakes at load time. A case that cannot measure what it claims to should fail before the run, not produce a green tick.

Trait Implementations

impl Clone for EvalCase

fn clone(&self) -> EvalCase

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EvalCase

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for EvalCase

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for EvalCase

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.