pub struct Tokenizer { /* private fields */ }Expand description
Tokenizer.
Implementations
sourceimpl Tokenizer
impl Tokenizer
sourcepub const fn new(dict: Dictionary) -> Self
pub const fn new(dict: Dictionary) -> Self
sourcepub fn ignore_space(self, yes: bool) -> Result<Self>
pub fn ignore_space(self, yes: bool) -> Result<Self>
Ignores spaces from tokens.
This option is for compatibility with MeCab. Enable this if you want to obtain the same results as MeCab.
Errors
VibratoError is returned when category SPACE is not defined in the input dictionary.
sourcepub fn max_grouping_len(self, max_grouping_len: usize) -> Self
pub fn max_grouping_len(self, max_grouping_len: usize) -> Self
Specifies the maximum grouping length for unknown words. By default, the length is infinity.
This option is for compatibility with MeCab.
Specifies the argument with 24 if you want to obtain the same results as MeCab.
Arguments
max_grouping_len: The maximum grouping length for unknown words. The default value is 0, indicating the infinity length.
sourcepub const fn dictionary(&self) -> &Dictionary
pub const fn dictionary(&self) -> &Dictionary
Gets the reference to the dictionary.
sourcepub fn new_worker(&self) -> Worker<'_>
pub fn new_worker(&self) -> Worker<'_>
Creates a new worker.
Auto Trait Implementations
impl RefUnwindSafe for Tokenizer
impl Send for Tokenizer
impl Sync for Tokenizer
impl Unpin for Tokenizer
impl UnwindSafe for Tokenizer
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more