Struct ReliabilityReport 

pub struct ReliabilityReport {
    pub metric: String,
    pub threshold: f64,
    pub k: usize,
    pub n_queries: usize,
    pub trials_per_query: usize,
    pub mean_pass_rate: f64,
    pub pass_at_k: f64,
    pub pass_all_k: f64,
    pub per_query: Vec<QueryReliability>,
}

Repeated-trial reliability report for a single metric.

This report turns a set of repeated MetricReports for the same metric into reliability estimates. Scores are thresholded into pass/fail outcomes first, then pass@k and pass^k are estimated per query and averaged.

use rig_retrieval_evals::{MetricReport, ReliabilityReport};

let trial_a = MetricReport::from_per_query(
    "recall@10".into(),
    vec![("q1".into(), 1.0), ("q2".into(), 0.0)],
);
let trial_b = MetricReport::from_per_query(
    "recall@10".into(),
    vec![("q1".into(), 1.0), ("q2".into(), 1.0)],
);

let reliability = ReliabilityReport::from_metric_reports(
    "recall@10",
    1.0,
    2,
    &[trial_a, trial_b],
)?;
assert_eq!(reliability.n_queries, 2);
assert_eq!(reliability.trials_per_query, 2);

Fields

metric: String

Metric identifier shared by every trial report.

threshold: f64

Score threshold used to convert each trial into pass/fail.

k: usize

Number of attempts sampled in pass@k / pass^k estimates.

n_queries: usize

Number of queries included in the reliability estimate.

trials_per_query: usize

Number of trials observed for each query.

mean_pass_rate: f64

Mean per-query pass rate.

pass_at_k: f64

Mean per-query pass@k.

pass_all_k: f64

Mean per-query pass^k.

per_query: Vec<QueryReliability>

Per-query reliability rows, in the first trial report’s query order.

Implementations

impl ReliabilityReport

pub fn from_metric_reports( metric: impl Into<String>, threshold: f64, k: usize, reports: &[MetricReport], ) -> Result<Self>

Build a repeated-trial reliability report from multiple MetricReports for the same metric.

Every report must contain the same query ids exactly once. k must be in 1..=reports.len(). Scores must be finite.

Trait Implementations

impl Clone for ReliabilityReport

fn clone(&self) -> ReliabilityReport

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ReliabilityReport

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for ReliabilityReport

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for ReliabilityReport

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.