Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(s: &str) -> Vec<String>
Expand description

Lowercase, split on non-alphanumerics, drop tokens shorter than 2 chars.