Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Split text on whitespace and ASCII punctuation; lowercase; keep tokens with at least 3 characters.