pub fn tokenize(source: &str) -> Vec<(Token<'_>, Span)>Expand description
Tokenize source code into a vector of (Token, Span) pairs.
This function:
- Runs the Logos lexer for fast tokenization
- Post-processes to detect indentation at line starts
- Handles lexer errors by producing Error tokens