pub fn tokenize(text: &str) -> Vec<String>
Tokenizes text into lowercase words by splitting on whitespace and non-alphanumeric characters, removes stop words, and applies stemming. Also splits camelCase/PascalCase identifiers.