Struct ress::Tokenizer

pub struct Tokenizer<'a> { /* fields omitted */ }

This structure will perform the low level tokenization before the Scanner provides additional context

Implementations

impl<'a> Tokenizer<'a>

pub fn new(stream: &'a str) -> Self

Create a new tokenizer using the provided string reference to create a JsBuffer

pub fn next(
    &mut self,
    allow_html_comment_close: bool
) -> Result<RawItem, RawError>

Get the next raw token from the js text

pub fn next_regex(&mut self, start_len: usize) -> Result<RawItem, RawError>

get the next regex token from the js text, providing the lenth of the already consumed token (this will be either 1 or 2)

note: this should only be used after first getting / or /= from the next method.

pub fn skip_whitespace(&mut self) -> (usize, usize)

Skip any whitespace that might be coming up