Struct ratel::tokenizer::Tokenizer

pub struct Tokenizer<'a> { /* fields omitted */ }

Methods

impl<'a> Tokenizer<'a>

fn new(source: &'a str) -> Self

fn get_token(&mut self) -> Result<Token>

fn read_template_kind(&mut self) -> Result<TemplateKind>

On top of being called when the opening backtick (`) of a template literal occurs, this method needs to be used by the parser while parsing a complex template string expression.

Note: Parser needs to expect a BraceClose token before calling this method to ensure that the tokenizer state is not corrupted.

fn asi(&self) -> bool

Check if Automatic Semicolon Insertion rules can be applied

fn invalid_token(&self) -> Error

fn read_regular_expression(&mut self) -> Result<Expression>