Skip to main content

tokenize_words

Function tokenize_words 

Source
pub fn tokenize_words(text: &str) -> Vec<&str>
Expand description

Simple word tokenizer that splits on non-alphanumeric characters.

Produces only non-empty tokens of alphanumeric characters.