Skip to main content

tokenize_words

Function tokenize_words 

Source
pub fn tokenize_words(s: &str) -> Vec<String>
Expand description

Word-and-punctuation tokenizer. Splits on whitespace, then separates leading/trailing punctuation into their own tokens.