Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Tokenise on Unicode word boundaries — anything that is not an alphanumeric scalar value (or _) splits the token. Lowercases each emitted token.