Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

A lexer for the MON language, also known as a tokenizer or scanner.

The Lexer’s primary role is to read MON source code as a stream of characters and break it down into a sequence of Tokens. Each token represents a meaningful unit of the language, like an identifier, a number, or a punctuation mark.

The Lexer is the first step in the compilation pipeline, providing the input for the Parser.

Example: How to use the Lexer

You can use the Lexer to tokenize a MON source string and inspect the tokens.

use mon_core::lexer::{Lexer, TokenType, Token};

let source = "{ key: 123 }";

// 1. Create a new lexer for the source code.
let mut lexer = Lexer::new(source);

// 2. Use `lex()` to get all tokens.
let tokens: Vec<Token> = lexer.lex();

assert_eq!(tokens[0].ttype, TokenType::LBrace);
assert_eq!(tokens[1].ttype, TokenType::Whitespace);
assert_eq!(tokens[2].ttype, TokenType::Identifier("key".to_string()));
assert_eq!(tokens[3].ttype, TokenType::Colon);
assert_eq!(tokens[4].ttype, TokenType::Whitespace);
assert_eq!(tokens[5].ttype, TokenType::Number(123.0));
assert_eq!(tokens[6].ttype, TokenType::Whitespace);
assert_eq!(tokens[7].ttype, TokenType::RBrace);
assert_eq!(tokens[8].ttype, TokenType::Eof);

// Alternatively, you can process tokens one by one using `next_token`.
let mut lexer = Lexer::new(source);
assert_eq!(lexer.next_token().ttype, TokenType::LBrace);
assert_eq!(lexer.next_token().ttype, TokenType::Whitespace);

Implementations

impl<'a> Lexer<'a>

pub fn new(input: &'a str) -> Self

Creates a new Lexer for the given input string.

pub fn lex(&mut self) -> Vec<Token>

Consumes the Lexer and returns a Vec<Token> containing all tokens from the source.

This method will tokenize the entire input string up to and including the final TokenType::Eof token.

pub fn next_token(&mut self) -> Token

Scans and returns the next Token from the input stream.

This is the core tokenizing function. When the end of the input is reached, it will repeatedly return a token of type TokenType::Eof.