Expand description
Compact lexer — token stream over UTF-8 source bytes.
Produces (SyntaxKind, &str) pairs with byte offsets, suitable
for consumption by the parser. Each Compact keyword has a dedicated
_KW SyntaxKind variant; literals and identifiers carry their
text verbatim.
Functions§
- lex
- Lex source code into a sequence of
(SyntaxKind, &str)pairs.