Struct nlprule::tokenizer::Tokenizer

pub struct Tokenizer { /* fields omitted */ }

The complete Tokenizer doing tagging, chunking and disambiguation.

Implementations

impl Tokenizer

pub fn new<P: AsRef<Path>>(p: P) -> Result<Self, Error>

Creates a new tokenizer from a path to a binary.

Errors

• If the file can not be opened.
• If the file content can not be deserialized to a rules set.

pub fn from_reader<R: Read>(reader: R) -> Result<Self, Error>

Creates a new tokenizer from a reader.

pub fn rules(&self) -> &Vec<DisambiguationRule>

pub fn tagger(&self) -> &Arc<Tagger>

pub fn chunker(&self) -> &Option<Chunker>

pub fn options(&self) -> &Arc<TokenizerOptions>

pub fn disambiguate<'t>(
    &'t self,
    tokens: Vec<IncompleteToken<'t>>
) -> Vec<DisambiguatedToken<'t>>

Apply rule-based disambiguation to the tokens. This does not change the number of tokens, but can change the content arbitrarily.

pub fn sentencize<'t>(&'t self, text: &'t str) -> Vec<Vec<IncompleteToken<'t>>>

Splits the text into sentences and tokenizes each sentence.

pub fn pipe<'t>(&'t self, text: &'t str) -> Vec<Vec<Token<'t>>>

Applies the entire tokenization pipeline including sentencization, tagging, chunking and disambiguation.

Trait Implementations

impl Default for Tokenizer

fn default() -> Tokenizer

Returns the "default value" for a type.

impl<'de> Deserialize<'de> for Tokenizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for Tokenizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.