Struct Tokenizer 

pub struct Tokenizer {
    pub dictionary: Dictionary,
    pub user_dictionary: Option<UserDictionary>,
    pub mode: Mode,
}

Tokenizer

Fields

dictionary: Dictionary

The dictionary to be used for tokenization.

user_dictionary: Option<UserDictionary>

The user dictionary to be used for tokenization. (Optional)

mode: Mode

The tokenization mode.

Implementations

impl Tokenizer

pub fn from_config(config: TokenizerConfig) -> LinderaResult<Self>

Create a new tokenizer from the tokenizer config.

Arguments

• config: The tokenizer config.

returns: LinderaResult

pub fn new( dictionary: Dictionary, user_dictionary: Option<UserDictionary>, mode: Mode, ) -> Self

Create a new tokenizer.

Arguments

• dictionary: The dictionary to be used for tokenization.
• user_dictionary: The user dictionary to be used for tokenization. (Optional)
• mode: The tokenization mode.

returns: LinderaResult

pub fn tokenize<'a>(&'a self, text: &'a str) -> LinderaResult<Vec<Token<'a>>>

Tokenize the text

Arguments

• text: The text to be tokenized.

returns: LinderaResult<Vec>

• Vec : The list of Token if succeeded
• LinderaError : Error message with LinderaErrorKind

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.