pub fn tokenize(text: &str) -> Vec<Range<usize>>Expand description
Split text into word, whitespace and punctuation tokens, as byte ranges.
A word is a run of alphanumerics and _; whitespace runs are one token;
every other character is its own token.