Struct rsjsonnet_lang::lexer::Lexer

pub struct Lexer<'a> { /* private fields */ }

The lexer. See the module-level documentation for more details.

Implementations

impl<'a> Lexer<'a>

pub fn new( str_interner: &'a StrInterner, span_mgr: &'a mut SpanManager, span_ctx: SpanContextId, input: &'a [u8] ) -> Self

Create a new lexer that operates on input.

pub fn lex_to_eof( self, whitespaces_and_comments: bool ) -> Result<Vec<Token>, LexError>

Lexes all the remaining input into a token Vec.

The last token will be an end-of-file token.

If whitespaces_and_comments is false, whitespace and comment tokens will be omitted from the output.

pub fn next_token(&mut self) -> Result<Token, LexError>

Lexes the next token.