Module ensemble 

Ensemble disagreement metrics for multi-model NER systems.

When running multiple NER models, disagreement patterns reveal:

• Ambiguous or difficult examples
• Model-specific biases
• Opportunities for ensemble voting

Key Metrics

• Agreement Rate: How often do models agree?
• Fleiss’ Kappa: Inter-rater reliability adjusted for chance
• Disagreement Entropy: Information content of disagreements
• Per-Entity-Type Agreement: Where do models disagree most?

Example

use anno_eval::eval::ensemble::{EnsembleAnalyzer, ModelPrediction};

let predictions = vec![
    ModelPrediction {
        model_name: "model_a".into(),
        entities: vec![("John".into(), "PER".into()), ("Google".into(), "ORG".into())],
    },
    ModelPrediction {
        model_name: "model_b".into(),
        entities: vec![("John".into(), "PER".into()), ("Google".into(), "LOC".into())],  // Disagrees on Google
    },
];

let analyzer = EnsembleAnalyzer::default();
let results = analyzer.analyze_single(&predictions);
println!("Agreement rate: {:.1}%", results.agreement_rate * 100.0);

Structs

DisagreementDetail

Details about a disagreement.

EnsembleAnalysisResults

Results of ensemble analysis across multiple examples.

EnsembleAnalyzer

Analyzer for ensemble model disagreements.

ModelPrediction

A single model’s predictions for one text.

SingleExampleAnalysis

Results of ensemble analysis for a single example.

Functions

agreement_grade

Grade agreement rate.

kappa_interpretation

Interpret Fleiss’ Kappa value.