Struct Lexer 

pub struct Lexer { /* private fields */ }

Lexer for tokenizing OxiLean source code.

Supports:

• Keywords and identifiers (including Unicode letters: Greek, math symbols)
• Number literals: decimal, hex (0x), binary (0b), octal (0o)
• Float literals: 3.14, 1.0e10, 1.5e-3
• String literals with enhanced escape sequences (\u{1234}, \x41, \0)
• Char literals: 'a', '\n', '\x41'
• Interpolated strings: s!"hello {name}"
• Doc comments: /-- ... -/ (block) and --- ... (line)
• Operators: >=, .., =>, <-, ->, etc.

Implementations

impl Lexer

pub fn new(input: &str) -> Self

Create a new lexer from source code.

pub fn next_token(&mut self) -> Token

Get the next token.

pub fn tokenize(&mut self) -> Vec<Token>

Tokenize the entire input.