pub fn tokenize(text: &str) -> Vec<String>Expand description
Split text into lowercase tokens.
CJK runs contribute both single characters and adjacent bigrams; a bigram is what actually discriminates in those scripts, while the unigrams keep recall for single-character terms.