Struct Lexer

pub struct Lexer<'a> { /* private fields */ }

A lexer (tokenizer) for a mathematical expression.

The Lexer programatically tokenizes one or more bytes of a program, with safety checks to enable notification that the program has concluded (EOF). Each Token is individually useless unless further processed; for instance, by the Parser.

As an example, an abstract representation of a program’s tokenization is shown:

in:
    123 + 281 ./ 3
    ^~~ ^ ^~~ ^~ ^
out (over 5 iterations):
    1. INTEGER(123)
    2. PLUS
    3. INTEGER(281)
    4. DIVIDEINT
    5. INTEGER(3)

We expect the Parser to be agnostic to the significance and relationship of each byte in the program, instead concerned only with the creation and typing of bytes. As such, the Lexer is built to refrain from analyzing anything but the values of bytes in a program.

Implementations

impl<'a> Lexer<'a>

pub fn from(text: &str) -> Lexer<'_>

Creates a Lexer from a text program.

pub fn skip(&mut self, amt: usize) -> &mut Lexer<'a>

Skips amt Tokens and returns the current Lexer. Primarily intended for use in method chaining:

let mut lexer = Lexer::from("2 + 3");
let result = lexer.skip(1).next_token().unwrap(); // => Token::PLUS

pub fn next_token(&mut self) -> Result<Token, ProgramError>

Tokenizes the next one or more characters in the program. If no valid Token can be made, the Lexer panics.

let mut lexer = Lexer::from("2 + 3");
let result = lexer.next_token().unwrap(); // => Token::NUMBER(2)