Struct AnalysisWorker 

pub struct AnalysisWorker { /* private fields */ }

A reusable tokenization session over a Tokenizer’s full analysis chain (character filters → segmentation → token filters), owning every per-call buffer: the Viterbi lattice and backtrace scratch (via SegmentWorker), the normalized-text buffer the character filters operate on, and the offset-mapping scratch.

Compared to Tokenizer::tokenize, repeated calls avoid the fresh lattice allocation, and — when character filters are configured — both the per-call String promotion of the input and the per-token surface String allocations (surfaces borrow the worker’s buffer instead).

Create a worker with Tokenizer::new_worker or Tokenizer::into_worker; it stays permanently bound to that tokenizer’s dictionary and filter chain. Returned tokens borrow the worker, so they must be consumed before the next call — the usual per-line loop compiles as-is:

let mut worker = tokenizer.new_worker();
for line in lines {
    let tokens = worker.tokenize(line)?;
    for token in &tokens {
        // use token.surface, ...
    }
}

The worker is Send + Sync; the &mut self API means cross-thread sharing requires external synchronization (one worker per thread, or a Mutex as lindera-binding-core does).

Implementations

impl AnalysisWorker

pub fn tokenize<'w>( &'w mut self, text: &'w str, ) -> LinderaResult<Vec<Token<'w>>>

Tokenizes text through the full analysis chain, reusing this worker’s internal buffers.

Produces exactly the same tokens as Tokenizer::tokenize for the same input and configuration. The returned tokens borrow both text and this worker, so they must be consumed before the next call.

引数

• text - The input text to tokenize.

戻り値

The filtered tokens, in reading order, with byte offsets corrected back to the original (pre-filter) text.

pub fn tokenize_nbest<'w>( &'w mut self, text: &'w str, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> LinderaResult<Vec<(Vec<Token<'w>>, i64)>>

Tokenizes text and returns the top-N results with costs, through the full analysis chain, reusing this worker’s internal buffers.

Produces exactly the same results as Tokenizer::tokenize_nbest for the same input and configuration.

引数

• text - The input text to tokenize.
• n - Maximum number of tokenizations to return.
• unique - Deduplicate results with identical word boundaries.
• cost_threshold - Discard paths costing more than best + threshold.

戻り値

Up to n (tokens, cost) pairs ordered by cost (best first).

pub fn set_mode(&mut self, mode: Mode)

Sets the segmentation mode for subsequent calls.

引数

• mode - The mode to use from the next call on.

pub fn set_keep_whitespace(&mut self, keep: bool)

Sets whether whitespace tokens are kept in the output for subsequent calls.

引数

• keep - true to keep whitespace tokens, false to drop them.

pub fn shrink_to(&mut self, text_len_hint: usize)

Immediately shrinks the worker’s internal buffers to what an input of text_len_hint bytes needs.

引数

• text_len_hint - Expected typical input length in bytes.

pub fn reset(&mut self)

Discards all internal buffers, replacing them with fresh ones.

Intended for recovery paths (e.g. after a panic poisoned a mutex holding this worker) where the buffers may be in an inconsistent intermediate state; configuration (dictionary, filters, mode) is preserved.