Skip to main content

lex

sage_parser

Function lex 

Source 
pub fn lex(source: &str) -> Result<LexResult, LexError>
Expand description

Lex source code into tokens.

This function tokenizes the entire source, collecting all errors rather than stopping at the first one. If any errors are found, they are all reported.

§Arguments

  • source - The source code to lex.

§Returns

  • Ok(LexResult) - Successfully lexed tokens with source reference.
  • Err(LexError) - One or more lex errors occurred.

§Errors

Returns LexError if the source contains invalid characters that cannot be tokenized. All errors are collected and reported together.

§Example

use sage_parser::{lex, Token};

let result = lex("let x = 42").unwrap();
assert_eq!(result.tokens()[0].token, Token::KwLet);