Trait alpino_tokenizer::Tokenizer

source · 
pub trait Tokenizer {
    // Required method
    fn tokenize(&self, text: &str) -> Option<Vec<Vec<String>>>;
}
Expand description

Tokenizer trait type.

Required Methods§

source

fn tokenize(&self, text: &str) -> Option<Vec<Vec<String>>>

Sentence-split and tokenize a paragraph of text.

The paragraph should be on a single line.

Implementors§