Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Lower-cased alphanumeric tokens. CJK characters become one token each so languages without spaces still get a usable keyword index.