Struct Trainer

pub struct Trainer<P> { /* private fields */ }

A trainer will build data about abbreviations, sentence starters, collocations, and context that tokens appear in. The data is used by the sentence tokenizer to determine if a period is likely part of an abbreviation, or actually marks the termination of a sentence.

Implementations

impl<P> Trainer<P>

where P: TrainerParameters + DefinesNonPrefixCharacters + DefinesNonWordCharacters,

pub fn new() -> Trainer<P>

Creates a new Trainer.

pub fn train(&self, doc: &str, data: &mut TrainingData)

Train on a document. Does tokenization using a WordTokenizer.