Struct CoreTokenizer 

pub struct CoreTokenizer { /* private fields */ }

Tokenizer that orchestrates tokenization on behalf of the bindings.

Internally holds a reusable AnalysisWorker behind a Mutex, so every call reuses the Viterbi lattice and scratch buffers instead of reallocating them (bindings keep one long-lived CoreTokenizer instance, which makes this the natural reuse point). Mutex — rather than RefCell — keeps CoreTokenizer: Send + Sync, which the Python (pyo3) and Node.js (napi) class wrappers require. All binding runtimes call tokenize while effectively single-threaded (GIL / one JS thread / request scope), so the lock is uncontended in practice.

Returns owned TokenViews so the bindings never handle borrowed lindera tokens directly.

Implementations

impl CoreTokenizer

pub fn from_segmenter( mode: &str, dictionary: Dictionary, user_dictionary: Option<UserDictionary>, ) -> CoreResult<Self>

Builds a tokenizer from segmenter parts, parsing mode from a string.

pub fn from_tokenizer(tokenizer: Tokenizer) -> Self

Wraps an already-built lindera Tokenizer, consuming it into the internal reusable worker (no user-dictionary copy).

pub fn tokenize(&self, text: &str) -> CoreResult<Vec<TokenView>>

Tokenizes text, returning owned TokenViews.

Reuses the internal worker’s buffers across calls; output is identical to tokenizing with a fresh lattice.

pub fn tokenize_surfaces(&self, text: &str) -> CoreResult<Vec<SurfaceView>>

Tokenizes text, returning surface-only views (SurfaceView) without materializing any morphological details.

This is the fast path for wakati-style callers: it skips the per-token details loading and string materialization that CoreTokenizer::tokenize pays (~14 allocations per IPADIC token down to ~1), while reusing the internal worker’s buffers like the other methods. Token filters configured on the tokenizer still run (some may load details internally); the saving on this path is the FFI-side materialization.

pub fn tokenize_nbest( &self, text: &str, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> CoreResult<Vec<(Vec<TokenView>, i64)>>

Tokenizes text and returns the N-best results as (tokens, cost) pairs.

Reuses the internal worker’s buffers across calls, like CoreTokenizer::tokenize.