Struct esr::lexer::Lexer

pub struct Lexer<'arena> {
    pub token: Token,
    pub quasi: &'arena str,
    /* private fields */
}

Fields

token: Token

Current Token from the source.

quasi: &'arena str

Implementations

impl<'arena> Lexer<'arena>

pub fn new(arena: &'arena Arena, source: &str) -> Self

Create a new Lexer from source using an existing arena.

pub unsafe fn from_ptr(ptr: *const u8) -> Self

Create a new Lexer from a raw pointer to byte string.

The source must be null terminated! Passing a pointer that is not null terminated is undefined behavior!

The source must be valid UTF8! Passing a pointer to data that is not valid UTF8 will lead to bugs or undefined behavior.

pub fn consume(&mut self)

Advances the lexer, produces a new Token and stores it on self.token.

pub fn token_as_str(&self) -> &'arena str

Create an &str slice from source spanning current token.

pub fn accessor_as_str(&self) -> &'arena str

Specialized version of token_as_str that crates an &str slice for the identifier following an accessor (.).

pub fn loc(&self) -> (u32, u32)

Get the start and end positions of the current token.

pub fn start(&self) -> u32

Get the start position of the current token.

pub fn end(&self) -> u32

Get the end position of the current token.

pub fn start_then_consume(&mut self) -> u32

Get the start position of the current token, then advance the lexer.

pub fn end_then_consume(&mut self) -> u32

Get the end position of the current token, then advance the lexer.

pub fn read_template_kind(&mut self)

On top of being called when the opening backtick (`) of a template literal occurs, this method needs to be used by the parser while parsing a complex template string expression.

Note: Parser needs to expect a BraceClose token before calling this method to ensure that the tokenizer state is not corrupted.

pub fn asi(&self) -> Asi

Get a definition of which ASI rules can be applied.

pub fn invalid_token(&mut self) -> Error

pub fn read_accessor(&mut self)

This is a specialized method that expects the next token to be an identifier, even if it would otherwise be a keyword.

This is useful when parsing member expressions such as foo.function, where function is actually allowed as a regular identifier, not a keyword.

The perf gain here comes mainly from avoiding having to first match the &str to a keyword token, and then match that token back to a &str.

pub fn read_regular_expression(&mut self) -> &'arena str