Module sampling 

Sampling strategies for NER evaluation.

Research Context

Random sampling introduces bias when entity types are imbalanced.

Problem	Effect	Solution
Type skew	High F1 on frequent types, low on rare	Stratified sampling
Seed sensitivity	Results vary wildly across seeds	Multiple seeds + variance
Dataset size	Small samples → high variance	Report confidence intervals

Stratified Sampling (Recommended)

Maintains proportional entity type distribution from the full dataset. Critical for domain-specific datasets where some types are rare.

Full dataset:    PER (60%), ORG (30%), LOC (10%)
Random sample:   PER (75%), ORG (20%), LOC (5%)   ← Biased!
Stratified:      PER (60%), ORG (30%), LOC (10%)  ← Representative

Example

use anno_eval::eval::sampling::stratified_sample;
use anno_eval::eval::datasets::GoldEntity;
use anno::EntityType;

let cases: Vec<(String, Vec<GoldEntity>)> = vec![
    ("John works at Apple".into(), vec![
        GoldEntity::new("John", EntityType::Person, 0),
        GoldEntity::new("Apple", EntityType::Organization, 14),
    ]),
    // ... more cases
];

// Sample 100 cases, maintaining entity type proportions
let sample = stratified_sample(&cases, 100, 42);

Functions

multi_seed_eval

Run evaluation with multiple seeds and aggregate variance.

stratified_sample

Stratified sampling maintaining entity type proportions.

stratified_sample_ner

Stratified sampling with entity type awareness.