Struct Lexer 

pub struct Lexer { /* private fields */ }

Character-by-character scanner producing tokens.

Implementations

impl Lexer

pub fn new(source: &str) -> Self

pub fn with_position(source: &str, line: usize, column: usize) -> Self

Create a lexer that starts counting from the given source position. Useful for re-lexing interpolated expressions at their original location.

pub fn tokenize_with_comments(&mut self) -> Result<Vec<Token>, LexerError>

Tokenize source code, including comment tokens.

pub fn tokenize(&mut self) -> Result<Vec<Token>, LexerError>