pub fn tokenize(text: &str) -> Vec<String>
Tokenizer matching phago-rag’s tokenizer.
Performs case-insensitive tokenization with stopword removal and minimum token length filtering.