Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(input: &str) -> Vec<Token>
Expand description

Single-pass tokenizer. Faithful port of the Java Tokenizer.tokenize. Whitespace is consumed and not emitted. Letter runs may include internal hyphens, apostrophes, underscores and stray ‘!’ when flanked by letters/digits.