Struct NERDataset 

pub struct NERDataset { /* private fields */ }

A collection of annotated NER examples with metadata and filtering.

NERDataset is the primary type for working with NER evaluation data. It wraps a Vec<AnnotatedExample> with convenient methods for filtering, statistics, and conversion to evaluation formats.

Example

use anno_eval::eval::dataset::{NERDataset, Domain};

// Load synthetic data
let mut dataset = NERDataset::synthetic();

// Filter to specific domain
let biomedical = dataset.filter_domain(Domain::Biomedical);

// Get statistics
let stats = biomedical.stats();
println!("Examples: {}, Entities: {}", stats.total_examples, stats.total_entities);

// Convert to test cases for evaluation
let test_cases = biomedical.to_test_cases();

Implementations

impl NERDataset

pub fn new(name: impl Into<String>) -> Self

Create an empty dataset with a name.

pub fn from_examples( name: impl Into<String>, examples: Vec<AnnotatedExample>, ) -> Self

Create a dataset from a vector of examples.

pub fn with_source(self, source: impl Into<String>) -> Self

Create a dataset with source information.

pub fn synthetic() -> Self

Load the full synthetic dataset (all domains, all difficulties).

This is the primary constructor for testing and development. The synthetic data covers many entity types and edge cases.

Example

use anno_eval::eval::dataset::NERDataset;

let dataset = NERDataset::synthetic();
assert!(!dataset.is_empty());

pub fn synthetic_domain(domain: Domain) -> Self

Load synthetic data for a specific domain.

pub fn from_json<P: AsRef<Path>>(path: P) -> Result<Self>

Load a dataset from a JSON file.

Supports both JSON arrays and JSONL (one object per line).

Format

[
  {
    "text": "John works at Google.",
    "entities": [
      {"text": "John", "label": "PER", "start": 0, "end": 4},
      {"text": "Google", "label": "ORG", "start": 14, "end": 20}
    ]
  }
]

pub fn from_conll<P: AsRef<Path>>(path: P) -> Result<Self>

Load a dataset from CoNLL-2003 format.

Format

Each line: word POS chunk NER-tag Empty lines separate sentences.

pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self>

Auto-detect format and load from file.

Tries CoNLL first, then JSON/JSONL.

pub fn name(&self) -> &str

Returns the dataset name.

pub fn source(&self) -> Option<&str>

Returns the source (file path, URL, etc.) if known.

pub fn len(&self) -> usize

Returns the number of examples.

pub fn is_empty(&self) -> bool

Returns true if the dataset is empty.

pub fn get(&self, index: usize) -> Option<&AnnotatedExample>

Get an example by index.

pub fn as_slice(&self) -> &[AnnotatedExample]

Returns a slice of all examples.

pub fn as_mut_slice(&mut self) -> &mut [AnnotatedExample]

Returns a mutable slice of all examples.

pub fn iter(&self) -> impl Iterator<Item = &AnnotatedExample>

Returns an iterator over the examples.

pub fn iter_mut(&mut self) -> impl Iterator<Item = &mut AnnotatedExample>

Returns a mutable iterator over the examples.

pub fn filter_domain(&self, domain: Domain) -> Self

Filter to a specific domain, returning a new dataset.

pub fn filter_difficulty(&self, difficulty: Difficulty) -> Self

Filter to a specific difficulty, returning a new dataset.

pub fn filter_entity_type(&self, entity_type: &EntityType) -> Self

Filter to examples containing a specific entity type.

pub fn filter<F>(&self, predicate: F) -> Self

where F: Fn(&AnnotatedExample) -> bool,

Filter using a custom predicate.

pub fn take(&self, n: usize) -> Self

Take the first n examples.

pub fn skip(&self, n: usize) -> Self

Skip the first n examples.

pub fn push(&mut self, example: AnnotatedExample)

Add an example to the dataset.

pub fn extend<I: IntoIterator<Item = AnnotatedExample>>(&mut self, iter: I)

Extend with examples from another source.

pub fn merge(&mut self, other: Self)

Merge another dataset into this one.

pub fn to_test_cases(&self) -> Vec<(String, Vec<GoldEntity>)>

Convert to test cases for evaluation functions.

This is the bridge to evaluate_ner_model() and similar functions.

pub fn into_test_cases(self) -> Vec<(String, Vec<GoldEntity>)>

Consume and convert to test cases.

pub fn into_inner(self) -> Vec<AnnotatedExample>

Convert to owned examples vec.

pub fn stats(&self) -> DatasetStats

Compute dataset statistics.

Trait Implementations

impl Clone for NERDataset

fn clone(&self) -> NERDataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NERDataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NERDataset

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for NERDataset

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Extend<AnnotatedExample> for NERDataset

fn extend<I: IntoIterator<Item = AnnotatedExample>>(&mut self, iter: I)

Extends a collection with the contents of an iterator.

fn extend_one(&mut self, item: A)

🔬This is a nightly-only experimental API. (extend_one)

Extends a collection with exactly one element.

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)

Reserves capacity in a collection for the given number of additional elements.

impl FromIterator<AnnotatedExample> for NERDataset

fn from_iter<I: IntoIterator<Item = AnnotatedExample>>(iter: I) -> Self

Creates a value from an iterator.

impl Index<usize> for NERDataset

type Output = AnnotatedExample

The returned type after indexing.

fn index(&self, index: usize) -> &Self::Output

Performs the indexing (container[index]) operation.

impl<'a> IntoIterator for &'a NERDataset

type Item = &'a AnnotatedExample

The type of the elements being iterated over.

type IntoIter = Iter<'a, AnnotatedExample>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl IntoIterator for NERDataset

type Item = AnnotatedExample

The type of the elements being iterated over.

type IntoIter = IntoIter<AnnotatedExample>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl Serialize for NERDataset

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.