tokenize

Function tokenize 

Source
pub fn tokenize(source: &str) -> Vec<(Token<'_>, Span)>
Expand description

Tokenize source code into a vector of (Token, Span) pairs.

This function:

  1. Runs the Logos lexer for fast tokenization
  2. Post-processes to detect indentation at line starts
  3. Handles lexer errors by producing Error tokens