Module annotator 

Multi-annotator disagreement modeling.

The Ground Truth Problem

NER annotation is inherently subjective. Different annotators disagree about:

• Entity boundaries: “New York City” vs “New York”
• Entity types: Is “Apple” a company or a product?
• What counts as an entity: Generic nouns? Metonymic references?

Traditional evaluation assumes a single gold standard, hiding this uncertainty. This module models annotator disagreement explicitly.

Disagreement Metrics

Metric	Description
Agreement Rate	% of tokens/spans all annotators agree on
Fleiss’ Kappa	Chance-corrected agreement for multiple annotators
Krippendorff’s Alpha	Agreement metric that handles missing data
Soft F1	F1 weighted by annotator agreement

Example

use anno_eval::eval::annotator::{MultiAnnotatorCorpus, AnnotatorAnalyzer};

let mut corpus = MultiAnnotatorCorpus::new();

// Add annotations from multiple annotators
corpus.add_annotation("doc1", "annotator_A", vec![
    ("Barack Obama", 0, 12, "PER"),
    ("Hawaii", 25, 31, "LOC"),
]);
corpus.add_annotation("doc1", "annotator_B", vec![
    ("Barack Obama", 0, 12, "PER"),
    ("Hawaii", 25, 31, "GPE"),  // Different type!
]);

let analyzer = AnnotatorAnalyzer::new(&corpus);
let stats = analyzer.compute_agreement();
println!("Span agreement: {:.2}%", stats.span_agreement * 100.0);
println!("Type agreement: {:.2}%", stats.type_agreement * 100.0);

Research Background

• Hirschman et al. (1998): “Automating Coreference” [cmp-lg/9803001]
  • Found only 16% of interannotator disagreements were genuine coreference disagreement
  • 84% were systematic errors (missed pronouns, overlooked chains, zone issues)
  • Two-stage annotation (markables first, linking second) can improve agreement
• Plank et al. (2014): “Learning part-of-speech taggers with inter-annotator agreement loss”
• Pavlick & Kwiatkowski (2019): “Inherent Disagreements in Human Textual Inferences”
• Uma et al. (2021): “Learning from Disagreement: A Survey”

Structs

AgreementStats

Agreement statistics across annotators.

AnnotatedDocument

Document with annotations from multiple annotators.

Annotation

A single annotation from one annotator.

AnnotatorAnalyzer

Analyzer for multi-annotator agreement.

ContentiousSpan

A span with significant annotator disagreement.

MultiAnnotatorCorpus

Corpus with multi-annotator annotations.

SoftEvaluator

Compute soft F1 that accounts for annotator disagreement.

SoftMetrics

Soft evaluation metrics.