Struct Lexer

pub struct Lexer<'a, 'p, 'ast> { /* private fields */ }

The lexer. See the module-level documentation for more details.

Implementations

impl<'a, 'p, 'ast> Lexer<'a, 'p, 'ast>

pub fn new( arena: &'p Arena, ast_arena: &'ast Arena, str_interner: &'a StrInterner<'p>, span_mgr: &'a mut SpanManager, span_ctx: SpanContextId, input: &'a [u8], ) -> Self

Create a new lexer that operates on input.

pub fn lex_to_eof( self, whitespaces_and_comments: bool, ) -> Result<Vec<Token<'p, 'ast>>, LexError>

Lexes all the remaining input into a token Vec.

The last token will be an end-of-file token.

If whitespaces_and_comments is false, whitespace and comment tokens will be omitted from the output.

pub fn next_token(&mut self) -> Result<Token<'p, 'ast>, LexError>

Lexes the next token.