Module multi_run 

Multi-run evaluation for statistical robustness.

Running evaluations multiple times with different random seeds or data shuffles provides confidence intervals and detects instability.

Research Context

Reporting mean ± standard deviation across multiple runs is a best practice from ML research. The Sommerschield (2023) survey on ancient languages notes that single-run results can be misleading due to:

• Random initialization effects
• Data ordering sensitivity
• Stochastic dropout/sampling

This module provides tools to run evaluations N times and aggregate results with proper statistical reporting.

Example

use anno_eval::eval::{MultiRunConfig, MultiRunEvaluator, MetricWithVariance};

let config = MultiRunConfig::new()
    .with_runs(5)
    .with_shuffle(true);

let evaluator = MultiRunEvaluator::new(config);
let results = evaluator.evaluate(&model, &test_cases)?;

println!("F1: {} (n={})", results.f1, results.f1.n);
// F1: 85.2% ± 1.3% (n=5)

Structs

ModelComparison

Result of comparing two models.

MultiRunConfig

Configuration for multi-run evaluation.

MultiRunEvaluator

Multi-run evaluator for NER models.

MultiRunResults

Results from multi-run evaluation.

Functions

compare_models_multi_run

Compare two models across multiple runs with statistical significance.