Skip to main content

Module token

Module token 

Source

Structs§

Token

Enums§

TokenKind

Functions§

tokenize
Single-pass tokenizer. Faithful port of the Java Tokenizer.tokenize. Whitespace is consumed and not emitted. Letter runs may include internal hyphens, apostrophes, underscores and stray ‘!’ when flanked by letters/digits.