Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(
    source: &str,
) -> Result<Vec<Spanned<Token>>, Vec<Spanned<LexerError>>>
Expand description

Tokenize source code into a vector of spanned tokens.

Skips whitespace and comments (unless you need them for formatting). Returns errors with their positions for nice error messages.

Handles:

  • Arithmetic: $((expr)) becomes Arithmetic("expr")
  • Here-docs: <<EOF\nhello\nEOF becomes HereDocStart + HereDoc("hello")