Struct vibrato::tokenizer::Tokenizer

pub struct Tokenizer { /* private fields */ }

Tokenizer.

Implementations

impl Tokenizer

pub const fn new(dict: Dictionary) -> Self

Creates a new instance.

Arguments

• dict: Dictionary to be used.

pub fn ignore_space(self, yes: bool) -> Result<Self>

Ignores spaces from tokens.

This option is for compatibility with MeCab. Enable this if you want to obtain the same results as MeCab.

Errors

VibratoError is returned when category SPACE is not defined in the input dictionary.

pub const fn max_grouping_len(self, max_grouping_len: usize) -> Self

Specifies the maximum grouping length for unknown words. By default, the length is infinity.

This option is for compatibility with MeCab. Specifies the argument with 24 if you want to obtain the same results as MeCab.

Arguments

• max_grouping_len: The maximum grouping length for unknown words. The default value is 0, indicating the infinity length.

pub const fn dictionary(&self) -> &Dictionary

Gets the reference to the dictionary.

pub fn new_worker(&self) -> Worker<'_>

Creates a new worker.