Crate gold_cmp 

gold-cmp

Pairwise comparison runner for gold-set LLM evals.

Pattern: for each gold case, ask a judge (LLM or human) which of two candidate outputs is better, or call it a tie. Aggregate into a win-rate with a 95% CI and a yes/no answer on “is A meaningfully better than B” using a two-sided binomial test (normal approximation; OK for n ≥ 30).

Example

use gold_cmp::{summarize, Outcome};
let outcomes = vec![
    Outcome::AWins, Outcome::AWins, Outcome::BWins,
    Outcome::Tie,   Outcome::AWins, Outcome::AWins,
    Outcome::BWins, Outcome::AWins,
];
let s = summarize(&outcomes);
assert_eq!(s.a_wins, 5);
assert!(s.a_win_rate_excl_ties > 0.5);

Structs

Summary

Aggregated summary.

Enums

Outcome

Outcome of one pairwise judgment.

Functions

summarize

Summarize a list of outcomes.