pub fn lex_all(source: &str) -> (Vec<Token>, Vec<LexerError>)Expand description
Lex an entire source file into a token vector upfront.
This is used by the parser to enable indexed token access instead of lazy lexing, which eliminates branching in the hot Pratt loop path.
Returns a tuple of (tokens, errors). The token vector is guaranteed to end with an Eof token, and includes a second Eof sentinel to make peek2 safe.