Struct CoreTokenizer 

pub struct CoreTokenizer { /* private fields */ }

Tokenizer that orchestrates tokenization on behalf of the bindings.

Wraps lindera::tokenizer::Tokenizer and returns owned TokenViews so the bindings never handle borrowed lindera tokens directly.

Implementations

impl CoreTokenizer

pub fn from_segmenter( mode: &str, dictionary: Dictionary, user_dictionary: Option<UserDictionary>, ) -> CoreResult<Self>

Builds a tokenizer from segmenter parts, parsing mode from a string.

pub fn from_tokenizer(tokenizer: Tokenizer) -> Self

Wraps an already-built lindera Tokenizer.

pub fn tokenize(&self, text: &str) -> CoreResult<Vec<TokenView>>

Tokenizes text, returning owned TokenViews.

pub fn tokenize_nbest( &self, text: &str, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> CoreResult<Vec<(Vec<TokenView>, i64)>>

Tokenizes text and returns the N-best results as (tokens, cost) pairs.