Structs§
Enums§
Functions§
- tokenize
- Single-pass tokenizer. Faithful port of the Java
Tokenizer.tokenize. Whitespace is consumed and not emitted. Letter runs may include internal hyphens, apostrophes, underscores and stray ‘!’ when flanked by letters/digits.