pub struct Tokenizer<'a> { /* private fields */ }
Expand description
This structure will perform the low level
tokenization before the Scanner
provides
additional context
Implementations§
source§impl<'a> Tokenizer<'a>
impl<'a> Tokenizer<'a>
sourcepub fn new(stream: &'a str) -> Self
pub fn new(stream: &'a str) -> Self
Create a new tokenizer using
the provided string reference
to create a JsBuffer
sourcepub fn next(
&mut self,
allow_html_comment_close: bool
) -> Result<RawItem, RawError>
pub fn next( &mut self, allow_html_comment_close: bool ) -> Result<RawItem, RawError>
Get the next raw token from the js text
sourcepub fn next_regex(&mut self, start_len: usize) -> Result<RawItem, RawError>
pub fn next_regex(&mut self, start_len: usize) -> Result<RawItem, RawError>
get the next regex token from the js text, providing the length of the already consumed token (this will be either 1 or 2)
note: this should only be used after first getting /
or /=
from the next
method.
sourcepub fn skip_whitespace(&mut self) -> (usize, usize)
pub fn skip_whitespace(&mut self) -> (usize, usize)
Skip any whitespace that might be coming up
Auto Trait Implementations§
impl<'a> RefUnwindSafe for Tokenizer<'a>
impl<'a> Send for Tokenizer<'a>
impl<'a> Sync for Tokenizer<'a>
impl<'a> Unpin for Tokenizer<'a>
impl<'a> UnwindSafe for Tokenizer<'a>
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more