Struct alpino_tokenizer::FiniteStateTokenizer

pub struct FiniteStateTokenizer { /* private fields */ }

Finite state tokenizer and sentence splitter.

This type implements a tokenizer based on a finite-state transducer. In principle, it should work with any tokenizing transducer built with the FSA utilities.

If Alpino’s transducer is used for tokenization, it is strongly recommended to use the AlpinoTokenizer data type. AlpinoTokenizer applies pre- and post-processing steps that are expected by Alpino’s transducer.

Implementations

impl FiniteStateTokenizer

pub fn from_buf_read<R>(read: R) -> Result<Self, TokenizerError>

where R: BufRead,

Trait Implementations

impl Tokenizer for FiniteStateTokenizer

fn tokenize(&self, text: &str) -> Option<Vec<Vec<String>>>

Sentence-split and tokenize a paragraph of text.