Struct Lexer

pub struct Lexer<'lexer> { /* private fields */ }

Lexer for tokenising input strings.

The Lexer provides methods for tokenising input strings into individual tokens. It supports various token types such as identifiers, numbers, strings, and operators. The Lexer uses a cursor-based approach to iterate over the input string and extract tokens.

Implementations

impl<'lexer> Lexer<'lexer>

pub fn new(input: &'lexer str) -> Self

Creates a new Lexer instance.

Arguments

• input - The input string to be tokenised.

pub fn next_token(&mut self) -> Token

Advances the lexer to the next token in the input stream and returns the token.

This function is essentially a large switch statement, containing branches corresponding to every token type. This function skips any whitespace and comments before identifying the next token. The token is represented by a Token struct, which contains information about its kind (e.g., identifier, operator, literal) and its span in the input stream.

Returns

The next token in the sequence, and will continue to return an Eof token once the end is reached.