pub fn tokenize(text: &str) -> Vec<String>Expand description
Split text into lowercased alphanumeric terms — the shared tokenizer for
both indexing and (per-token) querying. Deliberately minimal: Unicode
alphanumeric runs, lowercased, no stemming or stop-words, so behaviour is
predictable and dependency-free.