Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

Tokenizer over a byte slice.

Implementations

impl<'a> Lexer<'a>

pub fn new(data: &'a [u8]) -> Self

Creates a lexer at the start of data.

pub fn at(data: &'a [u8], pos: usize) -> Self

Creates a lexer positioned at byte offset pos.

pub fn pos(&self) -> usize

Current byte offset.

pub fn seek(&mut self, pos: usize)

Moves the cursor to byte offset pos.

pub fn next_token(&mut self) -> Result<Token>

Consumes and returns the next token.

pub fn peek_token(&mut self) -> Result<Token>

Returns the next token without consuming it.

pub fn skip_whitespace_and_comments(&mut self)

Advances past whitespace and % comments.

pub fn data(&self) -> &'a [u8]

The underlying input.