pub fn tokenize(
source: &str,
) -> Result<Vec<Spanned<Token>>, Vec<Spanned<LexerError>>>Expand description
Tokenize source code into a vector of spanned tokens.
Skips whitespace and comments (unless you need them for formatting). Returns errors with their positions for nice error messages.
Handles:
- Arithmetic:
$((expr))becomesArithmetic("expr") - Here-docs:
<<EOF\nhello\nEOFbecomesHereDocStart+HereDoc("hello")