Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Simple whitespace + lowercase tokenizer

For MVP, this is sufficient. Can be replaced with more sophisticated tokenizers (stemming, stopwords, etc.) later.