Struct Lexer 

pub struct Lexer<'src> { /* private fields */ }

A peekable wrapper around logos::Lexer that yields (Token, Span) pairs.

Internal design

The inner logos::Lexer is only advanced by read_next_token(), which adapts the tokenizer into the generic private peek cache. Lexer cannot read or take the cache slots directly; the cache fields are private to a nested module, and its public methods fill the first slot before returning or consuming it.

  cache = []             (initial / just consumed)
      │
      ▼  cache calls read_next_token()
  cache = [Token(_), ...] (token ready)
  cache.eof_seen = true   (EOF)
      │
      ▼  cache.next() takes the token
  cache = [...]           (remaining lookahead)

When the underlying logos::Lexer encounters an unrecognized character, the span of the first such character is recorded in first_error_span and the character is skipped. The parser surfaces this as a ParseError::UnknownToken when a top-level parse_* entry point finishes, regardless of whether the downstream parse happened to succeed.

Implementations

impl<'src> Lexer<'src>

pub fn new(source: &'src str) -> Self

pub fn peek(&mut self) -> Option<&Token>

Peek at the next token without consuming it.

pub fn peek_second(&mut self) -> Option<&Token>

Peek at the token after the next token without consuming either one.

pub fn peek_third(&mut self) -> Option<&Token>

Peek at the third token from the current position without consuming any token.

pub fn peek_with_span(&mut self) -> Option<(&Token, Span)>

Peek at the next token and its span without consuming it.

This delegates to the cache, which fills its first slot before returning a reference to it.

pub const fn first_error_span(&self) -> Option<Span>

Return the span of the first unrecognized character encountered during lexing.

Returns None if the lexer has not yet seen any invalid input. The value is set lazily as tokens are consumed; callers that need an up-to-date answer at a specific point should ensure lexing has progressed past the region of interest (e.g., by draining the remaining tokens).

pub fn next_token(&mut self) -> Option<(Token, Span)>

Consume and return the next token and its span.

The cache fills its first slot before taking from it, so this method cannot accidentally consume an uninitialized cache slot.

pub fn slice_at(&self, span: Span) -> &'src str

Get the source text corresponding to a span.

pub const fn source_len(&self) -> usize

Return the total length (in bytes) of the source string.

pub fn into_source_metadata(self) -> SourceMetadata