Module eval 

NER and Coreference evaluation framework.

Feature Gating

This module lives in the anno-eval crate (which is not intended for crates.io publishing right now) and intentionally pulls in a large amount of evaluation infrastructure: datasets, registries, harnesses, and reporting.

For a minimal dependency surface, prefer the anno crate without evaluation tooling. For evaluation workflows, depend on anno-eval (or use the anno CLI).

Relation to anno::eval

The anno crate also exposes a small anno::eval module behind the legacy eval feature (and the preferred alias analysis). That module contains only light, analysis-oriented primitives (e.g. coref metrics and cross-context clustering helpers). The full evaluation harness remains here in anno-eval::eval.

Overview

This module provides comprehensive evaluation tools for:

• Named Entity Recognition (NER): Standard metrics, error analysis, significance testing
• Coreference Resolution: MUC, B³, CEAF, LEA, BLANC, CoNLL F1

Why Multiple Coreference Metrics?

No single metric captures all aspects of coreference quality:

Metric	Measures	Blind to
MUC	Link recall/precision	Singletons, entity count
B³	Mention-level overlap	Link structure
CEAF	Optimal entity alignment	Within-cluster structure
LEA	Links weighted by entity size	Nothing (comprehensive)
BLANC	Rand index (coreference + non-coref)	Entity semantics

The CoNLL F1 (average of MUC, B³, CEAF-e) is the standard benchmark metric.

Metric Divergence: A Diagnostic Tool

When metrics disagree significantly, it reveals systematic model behaviors:

• High MUC, Low B³: Model makes too few links (conservative)
• Low MUC, High B³: Model over-clusters (aggressive)
• High CEAF variance: Inconsistent entity boundaries

Use MetricDivergence to quantify this and diagnose model behavior.

NER Evaluation

use anno_eval::eval::{evaluate_ner_model, GoldEntity, ErrorAnalysis};
use anno::RegexNER;

let model = RegexNER::new();
let test_cases = vec![
    ("Meeting on January 15".to_string(), vec![
        GoldEntity::new("January 15", anno::EntityType::Date, 11),
    ]),
];

let results = evaluate_ner_model(&model, &test_cases)?;
println!("F1: {:.1}%", results.f1 * 100.0);

Coreference Evaluation

use anno_eval::eval::{CorefChain, Mention, conll_f1, muc_score, b_cubed_score};

let gold = vec![
    CorefChain::new(0, vec![Mention::new("John", 0, 4), Mention::new("he", 20, 22)]),
];
let pred = gold.clone(); // Perfect match

let (p, r, f1) = conll_f1(&gold, &pred);
assert!((f1 - 1.0).abs() < 0.001);

Dataset Support

Dataset	Type	Size	Format
CoNLL-2003	NER	~22k sentences	BIO tags
WikiGold	NER	145 docs	CoNLL
WNUT-17	NER	~5k tweets	CoNLL
MultiNERD	NER	~50k examples	JSONL
GAP	Coref	4.5k examples	TSV
PreCo	Coref	12k docs	JSON

Metrics

NER Metrics:

• Precision, Recall, F1 (micro/macro)
• Per-entity-type breakdown
• Partial match (boundary overlap)
• Confidence threshold analysis

Coreference Metrics:

• MUC (link-based)
• B³ (mention-based)
• CEAF-e/m (entity/mention alignment)
• LEA (link-based entity-aware)
• BLANC (rand-index based)
• CoNLL F1 (average of MUC, B³, CEAF-e)

Error Analysis:

• Confusion matrix
• Error categorization (type, boundary, spurious, missed)
• Statistical significance testing (paired t-test)

Re-exports

pub use bio_adapter::BioScheme;

pub use modes::EvalConfig as ModeConfig;

pub use modes::EvalMode;

pub use datasets::GoldEntity;

pub use dataset_registry::AnnotationScheme;

pub use dataset_registry::DataFormat;

pub use dataset_registry::DatasetId as RegistryDatasetId;

pub use datasets::DatasetMetadata;

pub use loader::DatasetLoader;

pub use loader::LoadableDatasetId;

pub use loader::LoadedDataset;

pub use dataset::AnnotatedExample;

pub use dataset::DatasetStats;

pub use dataset::Difficulty;

pub use dataset::Domain;

pub use dataset::NERDataset;

pub use harness::BackendAggregateResult;

pub use harness::BackendDatasetResult;

pub use harness::BackendRegistry;

pub use harness::DatasetStatsSummary;

pub use harness::EvalConfig;

pub use harness::EvalHarness;

pub use harness::EvalResults;

pub use types::CorefDocStats;

pub use types::DocumentScale;

pub use types::GoalCheck;

pub use types::GoalCheckResult;

pub use types::LabelShift;

pub use types::MetricDivergence;

pub use types::MetricValue;

pub use types::MetricWithVariance;

pub use coref_loader::adversarial_coref_examples;

pub use coref_loader::synthetic_coref_dataset;

pub use coref_loader::CorefLoader;

pub use coref_loader::GapExample;

pub use book_scale::BookScaleAnalysis;

pub use book_scale::BookScaleAnalyzer;

pub use book_scale::BookScaleConfig;

pub use book_scale::BookScaleDiagnostics;

pub use book_scale::CorefEvalScores;

pub use book_scale::MetricReliability;

pub use book_scale::MultiBookReport;

pub use book_scale::PerBookEvaluation;

pub use book_scale::ReliabilityLevel;

pub use book_scale::Scores;

pub use book_scale::StratifiedEvaluation;

pub use book_scale::WindowedEvaluation;

pub use inter_doc_coref::InterDocCorefMetrics;

pub use cdcr::comprehensive_cdcr_dataset;

pub use cdcr::financial_news_dataset;

pub use cdcr::political_news_dataset;

pub use cdcr::science_news_dataset;

pub use cdcr::sports_news_dataset;

pub use cdcr::tech_news_dataset;

pub use cdcr::CDCRConfig;

pub use cdcr::CDCRMetrics;

pub use cdcr::CDCRResolver;

pub use cdcr::CrossDocCluster;

pub use cdcr::Document;

pub use cdcr::LSHBlocker;

pub use cdcr::MentionRef;

pub use discontinuous::evaluate_discontinuous_ner;

pub use discontinuous::DiscontinuousEvalConfig;

pub use discontinuous::DiscontinuousGold;

pub use discontinuous::DiscontinuousNERMetrics;

pub use discontinuous::TypeMetrics as DiscontinuousTypeMetrics;

pub use relation::evaluate_relations;

pub use relation::RelationEvalConfig;

pub use relation::RelationGold;

pub use relation::RelationMetrics;

pub use relation::RelationPrediction;

pub use relation::RelationTypeMetrics;

pub use advanced_evaluator::evaluator_for_task;

pub use advanced_evaluator::DiscontinuousEvaluator;

pub use advanced_evaluator::EvalResults as AdvancedEvalResults;

pub use advanced_evaluator::RelationEvaluator;

pub use advanced_evaluator::TaskEvaluator;

pub use visual::evaluate_visual_ner;

pub use visual::synthetic_visual_examples;

pub use visual::BoundingBox;

pub use visual::VisualEvalConfig;

pub use visual::VisualGold;

pub use visual::VisualNERMetrics;

pub use visual::VisualPrediction;

pub use visual::VisualTypeMetrics;

pub use advanced_harness::evaluate_discontinuous_gold_vs_gold;

pub use advanced_harness::evaluate_discontinuous_synthetic;

pub use advanced_harness::evaluate_relations_gold_vs_gold;

pub use advanced_harness::evaluate_relations_synthetic;

pub use advanced_harness::evaluate_visual_gold_vs_gold;

pub use advanced_harness::synthetic_dataset_stats;

pub use advanced_harness::AdvancedTaskResults;

pub use advanced_harness::ModelResult;

pub use advanced_harness::SyntheticDatasetStats;

pub use gender_bias::create_comprehensive_bias_templates;

pub use gender_bias::create_neopronoun_templates;

pub use gender_bias::create_winobias_templates;

pub use gender_bias::occupation_stereotype;

pub use gender_bias::GenderBiasEvaluator;

pub use gender_bias::GenderBiasResults;

pub use gender_bias::OccupationBiasMetrics;

pub use gender_bias::PronounGender;

pub use gender_bias::StereotypeType;

pub use gender_bias::WinoBiasExample;

pub use bias_config::BiasDatasetConfig;

pub use bias_config::DistributionValidation;

pub use bias_config::FrequencyWeightedResults;

pub use bias_config::StatisticalBiasResults;

pub use demographic_bias::create_diverse_location_dataset;

pub use demographic_bias::create_diverse_name_dataset;

pub use demographic_bias::DemographicBiasEvaluator;

pub use demographic_bias::DemographicBiasResults;

pub use demographic_bias::Ethnicity;

pub use demographic_bias::LocationExample;

pub use demographic_bias::LocationType;

pub use demographic_bias::NameExample;

pub use demographic_bias::NameFrequency;

pub use demographic_bias::NameResult;

pub use demographic_bias::Region;

pub use demographic_bias::RegionalBiasResults;

pub use demographic_bias::Script;

pub use temporal_bias::create_temporal_name_dataset;

pub use temporal_bias::Decade;

pub use temporal_bias::TemporalBiasEvaluator;

pub use temporal_bias::TemporalBiasResults;

pub use temporal_bias::TemporalGender;

pub use temporal_bias::TemporalNameExample;

pub use length_bias::create_length_varied_dataset;

pub use length_bias::EntityLengthEvaluator;

pub use length_bias::LengthBiasResults;

pub use length_bias::LengthBucket;

pub use length_bias::LengthTestExample;

pub use length_bias::WordCountBucket;

pub use calibration::calibration_grade;

pub use calibration::confidence_entropy;

pub use calibration::confidence_gap_grade;

pub use calibration::confidence_variance;

pub use calibration::CalibrationEvaluator;

pub use calibration::CalibrationResults;

pub use calibration::EntropyFilter;

pub use calibration::ReliabilityBin;

pub use calibration::ThresholdMetrics;

pub use robustness::robustness_grade;

pub use robustness::Perturbation;

pub use robustness::PerturbationMetrics;

pub use robustness::RobustnessEvaluator;

pub use robustness::RobustnessResults;

pub use ood_detection::ood_rate_grade;

pub use ood_detection::OODAnalysisResults;

pub use ood_detection::OODConfig;

pub use ood_detection::OODDetector;

pub use ood_detection::OODStatus;

pub use ood_detection::VocabCoverageStats;

pub use dataset_quality::check_leakage;

pub use dataset_quality::entity_imbalance_ratio;

pub use dataset_quality::DatasetQualityAnalyzer;

pub use dataset_quality::DifficultyMetrics;

pub use dataset_quality::QualityReport;

pub use dataset_quality::ReliabilityMetrics;

pub use dataset_quality::ValidityMetrics;

pub use learning_curve::suggested_train_sizes;

pub use learning_curve::CurveFitParams;

pub use learning_curve::DataPoint;

pub use learning_curve::LearningCurveAnalysis;

pub use learning_curve::LearningCurveAnalyzer;

pub use learning_curve::SampleEfficiencyMetrics;

pub use ensemble::agreement_grade;

pub use ensemble::kappa_interpretation;

pub use ensemble::DisagreementDetail;

pub use ensemble::EnsembleAnalysisResults;

pub use ensemble::EnsembleAnalyzer;

pub use ensemble::ModelPrediction;

pub use ensemble::SingleExampleAnalysis;

pub use dataset_comparison::compare_datasets;

pub use dataset_comparison::compute_stats;

pub use dataset_comparison::estimate_difficulty;

pub use dataset_comparison::DatasetComparison;

pub use dataset_comparison::DatasetStats as ComparisonStats;

pub use dataset_comparison::DifficultyEstimate;

pub use dataset_comparison::LengthStats;

pub use drift::ConfidenceDrift;

pub use drift::DistributionDrift;

pub use drift::DriftConfig;

pub use drift::DriftDetector;

pub use drift::DriftReport;

pub use drift::DriftWindow;

pub use drift::VocabularyDrift;

pub use active_learning::entities_to_candidates;

pub use active_learning::estimate_budget;

pub use active_learning::export_annotation_priority;

pub use active_learning::rank_for_annotation;

pub use active_learning::select_for_annotation;

pub use active_learning::ActiveLearner;

pub use active_learning::Candidate;

pub use active_learning::SamplingStrategy;

pub use active_learning::ScoreStats;

pub use active_learning::SelectionResult;

pub use error_analysis::EntityInfo;

pub use error_analysis::ErrorAnalyzer;

pub use error_analysis::ErrorCategory;

pub use error_analysis::ErrorInstance;

pub use error_analysis::ErrorPattern;

pub use error_analysis::ErrorReport;

pub use error_analysis::PredictedEntity;

pub use error_analysis::TypeErrorStats;

pub use threshold_analysis::format_threshold_table;

pub use threshold_analysis::interpret_curve;

pub use threshold_analysis::PredictionWithConfidence;

pub use threshold_analysis::ThresholdAnalyzer;

pub use threshold_analysis::ThresholdCurve;

pub use threshold_analysis::ThresholdPoint;

pub use report::BiasSummary;

pub use report::CalibrationSummary;

pub use report::CoreMetrics;

pub use report::DataQualitySummary;

pub use report::DemographicBiasMetrics;

pub use report::ErrorSummary;

pub use report::EvalReport;

pub use report::GenderBiasMetrics;

pub use report::LengthBiasMetrics;

pub use report::Priority;

pub use report::Recommendation;

pub use report::RecommendationCategory;

pub use report::ReportBuilder;

pub use report::SimpleGoldEntity;

pub use report::TestCase;

pub use report::TypeMetrics as ReportTypeMetrics;

pub use unified_evaluator::EvalMetadata;

pub use unified_evaluator::EvalSystem;

pub use unified_evaluator::UnifiedEvalResults;

pub use unified_evaluator::BiasEvalResults;

pub use unified_evaluator::CalibrationEvalResults;

pub use unified_evaluator::DataQualityEvalResults;

pub use unified_evaluator::StandardEvalResults;

pub use backend_name::BackendName;

pub use config_builder::BiasDatasetConfigBuilder;

pub use config_builder::TaskEvalConfigBuilder;

pub use few_shot::simulate_few_shot_task;

pub use few_shot::FewShotEvaluator;

pub use few_shot::FewShotGold;

pub use few_shot::FewShotPrediction;

pub use few_shot::FewShotResults;

pub use few_shot::FewShotTask;

pub use few_shot::FewShotTaskResults;

pub use few_shot::SupportExample;

pub use long_tail::format_long_tail_results;

pub use long_tail::EntityFrequency;

pub use long_tail::FrequencyBucket;

pub use long_tail::FrequencySplit;

pub use long_tail::LongTailAnalyzer;

pub use long_tail::LongTailResults;

pub use long_tail::TypePerformance;

pub use analysis::build_confusion_matrix;

pub use analysis::compare_ner_systems;

pub use analysis::ConfusionMatrix;

pub use analysis::ErrorAnalysis;

pub use analysis::ErrorType;

pub use analysis::NERError;

pub use analysis::NERSignificanceTest;

pub use sampling::multi_seed_eval;

pub use sampling::stratified_sample;

pub use sampling::stratified_sample_ner;

pub use evaluator::*;

pub use metrics::*;

pub use validation::*;

Modules

abstract_anaphora

Abstract anaphora evaluation infrastructure.

active_learning

Active learning utilities for NER annotation.

advanced_evaluator

Advanced evaluators for specialized NER tasks.

advanced_harness

Advanced evaluation harness for specialized NER tasks.

analysis

Error analysis and diagnostics for NER evaluation.

annotator

Multi-annotator disagreement modeling.

backend_factory

Backend Factory for Runtime Backend Creation

backend_name

Backend Name Enum

benchmark

Large-scale benchmark datasets for NER evaluation.

bias_config

Configuration and statistical utilities for bias evaluation.

bio_adapter

BIO tag sequence adapter.

book_scale

Book-Scale Coreference Diagnostics

bridging

Bridging Anaphora Resolution.

calibration

Confidence calibration metrics for NER evaluation.

cdcr

Cross-Document Coreference Resolution (CDCR).

cluster_encoder

Cross-context clustering primitives (shared via anno-metrics).

config_builder

Configuration Builder Pattern

coref

Coreference resolution data structures.

coref_loader

Coreference dataset loading and parsing.

coref_metrics

Coreference metrics (shared via anno-metrics).

coref_resolver

Coreference resolvers (shared via anno::metrics).

cross_context_eval

Cross-Context Coreference Evaluation Harness.

dataset

Dataset API for NER evaluation.

dataset_comparison

Dataset comparison for understanding distribution differences.

dataset_metadata

Table-driven dataset metadata.

dataset_quality

Dataset quality metrics for NER evaluation.

dataset_registry

Dataset Registry: Single Source of Truth

dataset_spec

Dataset specifications and metadata for NER evaluation.

datasets

Dataset loading for NER evaluation.

demographic_bias

Demographic bias evaluation for Named Entity Recognition.

discontinuous

Discontinuous NER evaluation metrics.

discourse_deixis

Discourse Deixis Resolution.

drift

Temporal drift detection for production NER monitoring.

ensemble

Ensemble disagreement metrics for multi-model NER systems.

error_analysis

Error analysis for NER systems.

evaluator

NER evaluation trait and implementations.

few_shot

Few-shot learning evaluation for NER.

gender_bias

Gender bias evaluation for coreference resolution.

harness

Evaluation harness for comprehensive NER benchmarking.

history

Evaluation history tracking with optional SQLite index.

incremental_coref

Incremental Coreference Resolution for Book-Scale Documents

inter_doc_coref

Evaluation metrics for inter-document coreference resolution.

learning_curve

Learning curve analysis for NER evaluation.

length_bias

Entity length bias evaluation for Named Entity Recognition.

loader

Dataset downloading and caching for NER evaluation.

long_tail

Long-tail entity evaluation for NER.

low_resource

Low-resource and morphologically complex language evaluation metrics.

metrics

Advanced evaluation metrics for NER.

modes

NER evaluation modes following SemEval-2013 Task 9.1.

multi_run

Multi-run evaluation for statistical robustness.

ner_metrics

Legacy NER Evaluation Metrics (MUC/SemEval-2013 standards).

neural_cluster_encoder

Neural Cluster Encoder for Cross-Context Coreference.

ood_detection

Out-of-Distribution (OOD) detection for NER systems.

prelude

Evaluation prelude - commonly used types for quick imports.

profiling

Profiling utilities for performance analysis.

ranking

Ranking evaluation metrics for Named Entity Disambiguation (NED).

relation

Relation extraction evaluation metrics.

report

Unified evaluation report.

robustness

Robustness testing for NER models.

sampling

Sampling strategies for NER evaluation.

shell_nouns

Shell Noun Resolution.

synthetic

Synthetic NER Test Datasets

synthetic_gen

Auto-generated synthetic test data.

task_evaluator

Comprehensive Task-Dataset-Backend Evaluation System

task_mapping

Task-Dataset-Backend Mapping System

temporal_bias

Temporal bias evaluation for Named Entity Recognition.

threshold_analysis

Confidence threshold analysis for NER systems.

types

Evaluation types: MetricValue, GoalCheckResult, etc.

unified_evaluator

Unified Evaluation System

validation

Validation utilities for NER evaluation.

visual

Visual NER evaluation metrics.

Structs

AggregateCorefEvaluation

Aggregate evaluation results across multiple documents.

CorefChain

A coreference chain: mentions that all refer to the same entity.

CorefChainStats

Statistics for stratified coreference evaluation.

CorefConfig

Configuration for SimpleCorefResolver.

CorefDocument

A document with coreference annotations.

CorefEvaluation

Complete coreference evaluation results.

CorefScores

Coreference evaluation scores (precision, recall, F1).

EvaluationMetadata

Additional evaluation metadata for reproducibility.

Mention

A single mention (text span) that may corefer with other mentions.

NEREvaluationResults

NER evaluation results.

SignificanceTest

Result of a paired significance test between two systems (very rough p-value approximation).

SimpleCorefResolver

Simple rule-based coreference resolver.

TypeMetrics

Per-entity-type metrics.

Enums

CorefMetric

Coreference evaluation metrics.

EvalTask

Evaluation task type.

Gender

Gender classification for NLP tasks.

MentionType

Type of referring expression in coreference.

Traits

CoreferenceResolver

Trait for coreference resolution algorithms.

Functions

b_cubed_score

B-cubed mention-based metric (Bagga & Baldwin, 1998).

blanc_score

BLANC Rand-index-style metric (Recasens & Hovy, 2010).

ceaf_e_score

CEAF entity-based metric (Luo, 2005), using the phi4 (Dice) similarity.

ceaf_m_score

CEAF mention-based metric (Luo, 2005), using the phi3 similarity function.

compare_ner_models

Compare multiple NER models on the same dataset.

compare_systems

Compare two systems using a paired t-test on CoNLL F1.

conll_f1

CoNLL F1: the official shared-task aggregate (Pradhan et al., 2012).

entity_type_matches

Entity type matching for evaluation.

entity_type_to_string

Convert EntityType to string label.

evaluate_ner_model

Evaluate NER model on a dataset.

evaluate_ner_model_with_mapper

Evaluate NER model with optional type normalization.

lea_score

LEA link-based entity-aware metric (Moosavi & Strube, 2016).

load_conll2003

Load CoNLL-2003 format dataset.

muc_score

MUC link-based metric (Vilain et al., 1995).