Struct SemanticTokenizer 

pub struct SemanticTokenizer { /* private fields */ }

A configurable tokenizer for semantic search indexing.

Supports whitespace splitting, word-boundary splitting, and character n-gram generation. Tokens can be lowercased, length-filtered, and stop-word filtered.

Implementations

impl SemanticTokenizer

pub fn new(config: TokenizerConfig) -> Self

Create a new tokenizer with the given configuration.

pub fn tokenize(&mut self, text: &str) -> Vec<Token>

Tokenize text according to the current mode and filters.

pub fn tokenize_batch(&mut self, texts: &[&str]) -> Vec<Vec<Token>>

Tokenize a batch of texts.

pub fn token_count(&self, text: &str) -> usize

Return the number of tokens that would be produced without allocating full Token structs.

pub fn add_stop_word(&mut self, word: &str)

Add a stop word.

pub fn remove_stop_word(&mut self, word: &str) -> bool

Remove a stop word. Returns true if it was present.

pub fn is_stop_word(&self, word: &str) -> bool

Check whether word is in the stop-word list.

pub fn normalize(&self, text: &str) -> String

Normalize text by applying lowercasing (if configured).

pub fn stats(&self) -> TokenizerStats

Return current tokenizer statistics.