Struct SegmentWorker 

pub struct SegmentWorker { /* private fields */ }

A reusable segmentation session that owns the Viterbi Lattice and the backtrace scratch buffer, so repeated calls avoid the per-call allocations that Segmenter::segment pays.

A worker is created from a Segmenter via Segmenter::new_worker or Segmenter::into_worker — never constructed directly — so the lattice is permanently tied to one dictionary. Reusing a lattice across dictionaries was the source of a past correctness bug (see lindera/tests/lattice_reuse_across_dictionaries.rs); this type makes that misuse unrepresentable.

Tokens returned by SegmentWorker::segment borrow the worker, so the borrow checker requires consuming (or cloning from) them before the next call — the usual per-line loop works without annotations:

let mut worker = segmenter.new_worker();
for line in lines {
    let tokens = worker.segment(line)?;
    for token in &tokens {
        // use token.surface, ...
    }
} // tokens dropped here; the next iteration may call segment again

The worker automatically bounds retained memory: one maximum-length sentence grows the lattice to several MB (~18 MB for 32 Ki ASCII slots, ~6 MB for a 32 KiB CJK sentence’s ~10.9 Ki slots), and without intervention that stays pinned for the worker’s lifetime. Every [SHRINK_WINDOW_CALLS] calls the worker compares the lattice capacity against the window’s largest sentence (with a [SHRINK_HYSTERESIS]x margin and a [SHRINK_FLOOR_SLOTS] floor) and shrinks it when oversized. SegmentWorker::shrink_to forces a shrink immediately.

The worker is Send + Sync (all fields are); the &mut self API means sharing one worker across threads requires external synchronization — the intended pattern is one worker per thread over a shared Segmenter.

Implementations

impl SegmentWorker

pub fn segment<'w>(&'w mut self, text: &'w str) -> LinderaResult<Vec<Token<'w>>>

Segments text reusing this worker’s internal buffers.

Produces exactly the same tokens as Segmenter::segment for the same input and configuration. The returned tokens borrow both text and this worker, so they must be consumed before the next call.

引数

• text - The input text to segment.

戻り値

The tokens segmented from text, in reading order.

pub fn segment_nbest<'w>( &'w mut self, text: &'w str, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> LinderaResult<Vec<(Vec<Token<'w>>, i64)>>

Segments text and returns the top-N results with costs, reusing this worker’s lattice.

Produces exactly the same results as Segmenter::segment_nbest for the same input and configuration.

引数

• text - The input text to segment.
• n - Maximum number of segmentations to return.
• unique - Deduplicate results with identical word boundaries.
• cost_threshold - Discard paths costing more than best + threshold.

戻り値

Up to n (tokens, cost) pairs ordered by cost (best first).

pub fn set_mode(&mut self, mode: Mode)

Sets the segmentation mode for subsequent calls.

The mode carries no lattice state (it is passed anew to every sentence), so switching per call is safe.

引数

• mode - The mode to use from the next call on.

pub fn set_keep_whitespace(&mut self, keep: bool)

Sets whether whitespace tokens are kept in the output for subsequent calls.

引数

• keep - true to keep whitespace tokens, false (MeCab-compatible default) to drop them.

pub fn set_max_grouping_len(&mut self, max_grouping_len: Option<usize>)

Sets the unknown-word grouping cap for subsequent calls (MeCab’s max-grouping-size semantics; None = unbounded, the default).

引数

• max_grouping_len - Maximum grouped characters beyond the first.

pub fn set_unknown_word_ladder(&mut self, unknown_word_ladder: bool)

Enables/disables the unknown-word length ladder for subsequent calls (see Segmenter::unknown_word_ladder; defaults to true).

引数

• unknown_word_ladder - Whether to emit the length ladder.

pub fn set_space_penalty( &mut self, space_penalty: Option<SpacePenaltyConfig>, ) -> LinderaResult<()>

Sets the left-space penalty rules for subsequent calls (see Segmenter::space_penalty; None disables it, overriding the rules the dictionary ships and applies by default).

Rebuilds the per-word-id lookup, so prefer setting this once.

引数

• space_penalty - The rules, or None to disable the penalty.

戻り値

Ok(()), or an error when the dictionary schema has no part-of-speech field; the previous setting is kept on error.

pub fn set_space_penalty_from_dictionary(&mut self) -> LinderaResult<()>

Enables the left-space penalty with the rules the dictionary ships in its metadata (see Segmenter::space_penalty_from_dictionary), e.g. to restore the default after set_space_penalty(None).

戻り値

Ok(()), or an error when the dictionary ships no rules; the previous setting is kept on error.

pub fn segmenter(&self) -> &Segmenter

Returns a shared reference to the underlying segmenter.

No &mut accessor is provided on purpose: swapping the dictionary out from under the reused lattice would break the dictionary↔lattice pairing this type exists to guarantee.

戻り値

The segmenter this worker is bound to.

pub fn shrink_to(&mut self, text_len_hint: usize)

Immediately shrinks the worker’s internal buffers to what an input of text_len_hint bytes needs, and resets the automatic-shrink window.

The lattice is slot(character)-denominated (#943); a byte length is a valid conservative bound, since a sentence never has more characters than bytes.

引数

• text_len_hint - Expected typical input length in bytes.

pub fn reset(&mut self)

Discards the internal buffers, replacing them with fresh ones.

Intended for recovery paths (e.g. after a panic poisoned a mutex holding this worker) where the buffers may hold an inconsistent intermediate state; the segmenter configuration is preserved.