Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

A standalone lexer that produces tokens on demand without heap allocation.

The lexer processes input bytes and yields Token values one at a time. It handles whitespace/comment skipping, number parsing, string escape sequences, character literals, and symbol case-folding.

No-std Compatible

The lexer uses fixed-size stack buffers for string and symbol content, avoiding any heap allocation. String literals are limited to 1024 characters and symbols to 64 characters.

Example

use grift_parser::Lexer;

let mut lexer = Lexer::new("(+ 1 2)");
while let Some(result) = lexer.next_token() {
    let spanned = result.unwrap();
    // Process token...
}

Implementations

impl<'a> Lexer<'a>

pub fn new(input: &'a str) -> Lexer<'a>

Create a new lexer from a string slice

pub fn from_bytes(input: &'a [u8]) -> Lexer<'a>

Create a new lexer from a byte slice

pub fn loc(&self) -> SourceLoc

Get the current source location

pub fn symbol_bytes(&self, len: usize) -> &[u8]

Get the lowercased symbol bytes from the last Token::Symbol produced.

The len field from Token::Symbol { len } indicates how many bytes are valid.

pub fn string_chars(&self, len: usize) -> &[char]

Get the string characters from the last Token::String produced.

The len field from Token::String { len } indicates how many characters are valid.

pub fn position(&self) -> (usize, usize)

Get current position in the input

pub fn has_more(&mut self) -> bool

Check if there’s more input (after skipping whitespace/comments)

pub fn next_token(&mut self) -> Option<Result<SpannedToken, LexError>>

Produce the next token, or None if input is exhausted.

Returns None when there is no more input (after whitespace). Returns Some(Err(...)) for lexer errors. Returns Some(Ok(...)) for successfully lexed tokens.