Struct Lexer 

pub struct Lexer<'src> { /* private fields */ }

A stream-based lexical analyzer capable of interpreting string sources.

Lexer sequentially reads the underlying string slice and produces tokens on demand via the next() and peek() methods.

Implementations

impl<'src> Lexer<'src>

pub fn new(source: &'src str) -> Self

Creates a new Lexer given a source string slice.

pub fn with_keywords(self, keywords: &[&'src str]) -> Self

Configures the lexer with a set of predefined keywords to recognize.

pub fn next(&mut self) -> Token

Returns the next token in the stream, consuming it in the process. If an EOF is reached, it will continue to return EOF tokens.

pub fn peek(&mut self) -> &Token

Returns a reference to the next token without consuming it. Subsequent calls to peek() or next() will return this same token.