Module loader 

Dataset downloading and caching for NER evaluation.

Downloads, caches, and parses real NER datasets from public sources. Follows burntsushi’s philosophy: real-world data, not toy examples.

Quick Start

use anno_eval::eval::{DatasetLoader, LoadableDatasetId};

let loader = DatasetLoader::new()?;
let dataset = loader.load(LoadableDatasetId::WikiGold)?;
println!("Loaded {} sentences", dataset.len());

Dataset IDs: catalog vs loadable

The full dataset catalog lives in dataset_registry::DatasetId. Not every dataset in the catalog can be downloaded/parsed by this crate (some require licenses or have unimplemented formats).

This module provides LoadableDatasetId, a wrapper that guarantees a dataset has a loading implementation. Use it for DatasetLoader::{load, load_or_download}.

Supported Datasets

Dataset	Source	License	Entities
WikiGold	Wikipedia	CC-BY	PER, LOC, ORG, MISC
WNUT-17	Social Media	Open	person, location, corporation, etc.
MIT Movie	MIT	Research	actor, director, genre, title, etc.
MIT Restaurant	MIT	Research	amenity, cuisine, dish, etc.
CoNLL-2003 Sample	Public	Research	PER, LOC, ORG, MISC
OntoNotes Sample	Public	Research	18 entity types
BC5CDR	PubMed	Research	Disease, Chemical
NCBI Disease	PubMed	Research	Disease

Design Philosophy

• Lazy downloading: Only fetch what’s needed
• Persistent caching: Never re-download unchanged data
• Integrity verification: SHA256 checksums for all downloads
• Graceful degradation: Work offline with cached data
• Clear errors: Explain exactly what went wrong

Extended Example

use anno_eval::eval::{DatasetLoader, LoadableDatasetId};

let loader = DatasetLoader::new()?;

// Check cache status before loading
if loader.is_cached(LoadableDatasetId::WikiGold) {
    println!("WikiGold is cached, will load from disk");
}

// Load dataset (downloads if not cached, verifies checksum)
let dataset = loader.load(LoadableDatasetId::WikiGold)?;
println!("Loaded {} sentences with {} entities",
    dataset.len(), dataset.entity_count());

// Iterate over examples
for example in dataset.iter() {
    println!("Text: {}", example.text);
    for entity in &example.entities {
        println!("  {} [{}]", entity.text, entity.entity_type);
    }
}

Re-exports

pub use super::dataset_registry::DatasetId;

Structs

AnnotatedSentence

Annotated sentence with tokens.

AnnotatedToken

Annotated token with NER tag.

CacheManifest

Cache manifest tracking all downloaded datasets.

CacheManifestEntry

Entry in the cache manifest tracking a downloaded dataset file.

DatasetLoader

Loads and caches NER datasets.

DatasetMetadata

Dataset metadata for provenance and reproducibility.

DatasetStats

Dataset statistics.

LoadableDatasetId

Dataset identifier guaranteed to be loadable by DatasetLoader.

LoadedDataset

A loaded dataset with metadata.

RelationDocument

A document with text and gold relations for relation extraction evaluation.

TemporalMetadata

Temporal metadata for datasets (optional).

Enums

DataSource

Where the dataset was loaded from.