Trait Tokenizer

pub trait Tokenizer: Send + Sync {
    // Required method
    fn tokenize(&self, text: &str, output: &mut Vec<Token>);
}

Expand description

Breaks input text into a sequence of tokens.

Implementations must be thread-safe (Send + Sync) so that analyzers can be shared across indexing threads.

Required Methods§

Tokenize text, appending tokens to output.

The output buffer is caller-owned and reusable — callers may pre-allocate it and clear between calls to avoid repeated allocation.

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".