Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(task: &str) -> Vec<String>
Expand description

Tokenize a free-form task into lowercased search terms: split on non-identifier characters, then split camelCase/snake_case, dropping stopwords and 1-character noise.