Struct Lexer 

pub struct Lexer<'a, T> {
    pub source: &'a Source<'a>,
    pub position: usize,
    pub current_char: Option<char>,
    /* private fields */
}

Tokenizes the source code.

Fields

source: &'a Source<'a>

The source code to be tokenized.

position: usize

The current position in the source code.

current_char: Option<char>

The current character being processed.

Implementations

impl<'a, T> Lexer<'a, T>

pub fn new( source: &'a Source<'a>, rules: Vec<Box<dyn LexerRule<'a, T>>>, ) -> Self

Creates a new Lexer instance with the given source code and rules.

pub fn advance(&mut self)

Advances the lexer to the next character in the source code.

pub fn jump_to(&mut self, position: usize)

Jumps to a specific position in the source code.

pub fn get_token(&mut self) -> Result<Option<Token<T>>, Error<'_>>

Attempts to get the next token from the lexer using the defined rules.

If a token is found, it returns Ok(Some(token)). If no token is found, it returns Ok(None). If an error occurs, it returns Err(error).