Struct Lexer 

pub struct Lexer {
    pub source: Source,
    pub tokens: Vec<Token>,
    pub position: Position,
}

The lexer is responsible for converting the source code into a list of tokens.

Fields

source: Source

tokens: Vec<Token>

position: Position

Implementations

impl Lexer

pub fn new(source: Source) -> Lexer

Create a new lexer from a source string.

Arguments

• source - An instance of Source containing the source code.

Example

use roan_ast::{Lexer, TokenKind};
use roan_ast::source::Source;
let source = Source::from_string("let x = 10;".to_string());
let mut lexer = Lexer::new(source);
let tokens = lexer.lex().expect("Failed to lex source code");

assert_eq!(tokens.first().unwrap().kind, TokenKind::Let);

impl Lexer

pub fn lex(&mut self, lex_comments: bool) -> Result<Vec<Token>, Error>

Lex the source code and return a list of tokens.

During the lexing process, the lexer will consume the source code character by character and convert it into a list of tokens. The lexer will skip whitespace and comments.

When EOF is reached, the lexer will return the list of tokens.

pub fn is_eof(&self) -> bool

Check if the lexer has reached the end of the source code.

pub fn current(&mut self) -> Option<char>

Get the current character in the source code.

pub fn consume(&mut self) -> Option<char>

Consume the current character and move to the next one.

pub fn is_identifier_start(&self, c: char) -> bool

Check if the character is a valid identifier start character.

pub fn is_number_start(&self, c: char) -> bool

Check if the character is a valid number start character.

pub fn peek(&self) -> Option<char>

Peek at the next character in the source code.

pub fn next_token(&mut self) -> Result<Option<Token>, Error>

Get the next token in the source code.

pub fn consume_number(&mut self) -> (NumberType, String)

Consume a number.

Can be either an integer or a float.

pub fn parse_char(&mut self) -> Result<char, Error>

Parses a character literal. Throws an error if more than one character is found.

pub fn lex_potential_double( &mut self, expected: char, one_char: TokenKind, double_char: TokenKind, ) -> TokenKind

pub fn lex_potential_triple( &mut self, expected: char, one_char: TokenKind, double_char: TokenKind, triple_char: TokenKind, ) -> TokenKind

pub fn match_next(&mut self, ch: char) -> bool

Check if the next character matches the given character.