pub fn tokenize(input: &str) -> Vec<Token>Expand description
Single-pass tokenizer. Faithful port of the Java Tokenizer.tokenize.
Whitespace is consumed and not emitted. Letter runs may include internal hyphens,
apostrophes, underscores and stray ‘!’ when flanked by letters/digits.