Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Split text into lowercase tokens.

CJK runs contribute both single characters and adjacent bigrams; a bigram is what actually discriminates in those scripts, while the unigrams keep recall for single-character terms.