pub fn lex(source: &str) -> Vec<(SyntaxKind, &str)>Expand description
Lex source into a sequence of (SyntaxKind, slice) pairs.
Every byte of source appears in exactly one token (lossless).
The only mutable state is a string_depth: u32 tracking nested string
interpolation. When string_depth > 0, we are inside a string and lex
STRING_TEXT/STRING_ESCAPE/QUOTE/L_BRACE. { inside a string
increments depth and exits string mode; } outside a string with
depth > 0 decrements depth and re-enters string mode.