Struct Lexer 

pub struct Lexer<'lex> { /* private fields */ }

Produces Tokens from an input string.

Implementations

impl<'lex> Lexer<'lex>

pub fn new(input: &str, offset: BytePos) -> Lexer<'_>

Creates a new Lexer to read tokens from the input string.

pub fn next_token(&mut self) -> Result<(Span, Token<'lex>), ParseError>

Scans the input stream for the next token, returning the token and the span of input text from which it was scanned.

pub fn skip_shebang(&mut self)

Skips over a shebang line at the start of input. This is used when parsing files which, on Unix systems, may use a line consisting of #! followed by a path to the interpreter.

Panics

Panics if any tokens have already been scanned from the input stream.

pub fn current_position(&self) -> BytePos

Returns the current position within the input string.

The next call to next_token will begin searching at this point.