pub fn tokenize(text: &str) -> Vec<String>
Lowercase alphanumeric tokens; everything else separates. Deliberately simple (no stemming, no stopwords) – linguistic layers stack on later without changing the keyspace.