Struct delta_inc::lex::Lexer

pub struct Lexer<T: Tokenizer> { /* private fields */ }

Provides machinery for splitting up an underlying sequence of items into a sequence of tokens, where each token can correspond to one or more items in the underlying sequence.

Implementations

impl<T: Tokenizer> Lexer<T>

pub fn new(input: Vec<T::Item>, tokeniser: T) -> Self

Construct a new lexer for a given string slice.

pub fn offset(&self) -> usize

Determine the current lexing position

pub fn reset(&mut self, offset: usize)

Reset the lexing position to a given (known) offset.

pub fn is_eof(&self) -> bool

Check whether the lexer has reached the end of the file or not.

pub fn get(&self, span: Span<T::Token>) -> &[T::Item]

Get the slice which corresponds to a given span from the underlying sequence.

pub fn peek(&self) -> Span<T::Token>

Peek at the next token in the sequence, or none if we have reached the end.

pub fn next(&mut self) -> Span<T::Token>

Get the next token in the sequence, or none if we have reached the end.

pub fn snap(&mut self, kind: T::Token) -> Result<T::Token>

Match a given token type in the current stream. If the kind matches, then the token stream advances. Otherwise, it remains at the same position and an error is returned.

pub fn snap_any(&mut self, kinds: &[T::Token]) -> Result<T::Token>

Match a given token type in the current stream for a set of candidates. If one of the candidates matches, then the token stream advances. Otherwise, it remains at the same position and an error is returned.