Struct Scorecard 

pub struct Scorecard {

    pub model: String,
    pub provider: String,
    pub total: usize,
    pub passed: usize,
    pub passed_any: Option<usize>,
    pub runs_per_case: usize,
    pub check_pass_rate: f64,
    pub malformed_tool_args: u32,
    pub unknown_tools: u32,
    pub tool_errors: u32,
    pub runs_errored: usize,
    pub mean_turns: f64,
    pub median_latency_ms: u64,
    pub total_usage: Usage,
    pub wall_clock_ms: u64,
    pub by_tag: Vec<TagScore>,
}

Aggregate view of one model’s run over the whole case set.

When each case ran more than once (runs_per_case > 1), total counts cases, and passed counts cases that passed every run — pass^k, the reliability number. Reliability decays much faster than mean success (τ-bench measured 61% pass^1 falling under 25% by pass^8), and a scorecard reporting only the mean hides exactly that. passed_any (pass@k, the capability number) is kept beside it; the gap between the two is the model’s unreliability, made visible. With one run per case the two coincide and everything reads as it always did.

Fields

model: String

provider: String

total: usize

Distinct cases, regardless of how many times each ran.

passed: usize

Cases that passed every run — pass^k.

passed_any: Option<usize>

Cases that passed at least one run — pass@k. None on single-run scorecards (it would merely repeat passed), which is also what keeps reports written before --runs existed loading unchanged.

runs_per_case: usize

How many times each case ran.

check_pass_rate: f64

Checks passed / checks attempted, over every run. Partial credit, unlike passed.

malformed_tool_args: u32

unknown_tools: u32

tool_errors: u32

runs_errored: usize

mean_turns: f64

median_latency_ms: u64

Median is the honest latency number here — one 900s timeout would dominate a mean and tell you nothing about typical behaviour.

total_usage: Usage

wall_clock_ms: u64

by_tag: Vec<TagScore>

Implementations

impl Scorecard

pub fn of( graded: &[GradedCase], model: String, provider: String, wall_clock_ms: u64, ) -> Self

pub fn pass_rate(&self) -> f64

Trait Implementations

impl Clone for Scorecard

fn clone(&self) -> Scorecard

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Scorecard

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Scorecard

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Scorecard

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.