Skip to main content

simple_tokenize

Function simple_tokenize 

Source
pub fn simple_tokenize(text: &str) -> Vec<String>
Expand description

Simple tokenizer for demo purposes - splits on whitespace and punctuation