Type Alias lindera::Tokenizer

source · 
pub type Tokenizer = Tokenizer;

Aliased Type§

struct Tokenizer {
    pub dictionary: Dictionary,
    pub user_dictionary: Option<UserDictionary>,
    pub mode: Mode,
}

Fields§

§dictionary: Dictionary

The dictionary to be used for tokenization.

§user_dictionary: Option<UserDictionary>

The user dictionary to be used for tokenization. (Optional)

§mode: Mode

The tokenization mode.