pub fn tokenize(text: &str) -> Vec<String>Expand description
tokenize(text): split on runs of non-[0-9A-Za-z] (every non-ASCII char
is a separator), split each piece at ASCII lowercase→uppercase seams, then
casefold (pure-ASCII pieces: exactly A-Z -> a-z).