pub struct Lexer<'src> { /* private fields */ }Expand description
the scanner state: the source, its bytes (for cheap ASCII peeking), the current byte offset, and the mode stack.
Implementations§
Source§impl<'src> Lexer<'src>
impl<'src> Lexer<'src>
Sourcepub fn tokenize(src: &'src str) -> Result<Vec<Token>, QalaError>
pub fn tokenize(src: &'src str) -> Result<Vec<Token>, QalaError>
scan src into a token stream ending in TokenKind::Eof, or the first
lex error.
fail-fast: the first malformed token aborts with its QalaError. an empty,
whitespace-only, or comment-only source yields exactly [Eof] and no error.
a leading UTF-8 BOM is skipped silently.
Auto Trait Implementations§
impl<'src> Freeze for Lexer<'src>
impl<'src> RefUnwindSafe for Lexer<'src>
impl<'src> Send for Lexer<'src>
impl<'src> Sync for Lexer<'src>
impl<'src> Unpin for Lexer<'src>
impl<'src> UnsafeUnpin for Lexer<'src>
impl<'src> UnwindSafe for Lexer<'src>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more