pub fn tokenize(text: &str) -> Vec<String>Expand description
Tokenizes text to match index rules and queries.
Splits text into tokens using regex pattern, normalizes each token (lowercase), and filters out empty strings and stopwords.
ยงReturns
A vector of token strings.