Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(text: &str) -> Vec<String>
Expand description

Split text into lowercased alphanumeric words (length >= 2). find_text semantics are substring-based, so query tokens are treated the same way here and candidates are later verified with the exact predicate.