pub fn lex(source: &str) -> Vec<(SyntaxKind, &str)>Expand description
Lex source into a sequence of (SyntaxKind, slice) pairs.
Every byte of source appears in exactly one token — this is the
lossless-roundtrip invariant the CST and its proptests depend on. The
only mutable state is a string_depth: u32 tracking nested string /
interpolation regions, mirroring brink-syntax’s lexer: a " toggles
string-scanning mode regardless of whether the surrounding context turns
out to be an expression string literal or literal dialogue-quote prose
text — the parser, not the lexer, decides which node shape a quoted
run gets, from its structural position (same one-token-stream,
position-decides-shape pattern brace_scan uses in the ink parser).