pub struct Tokenizer<'a> { /* private fields */ }text only.Expand description
Implementations§
Source§impl<'a> Tokenizer<'a>
impl<'a> Tokenizer<'a>
Sourcepub fn new(input: &'a str) -> Self
pub fn new(input: &'a str) -> Self
Create a tokenizer positioned at the start of input.
A leading UTF-8 byte-order mark (U+FEFF) is silently skipped — some
editors prepend one when saving, and it is not a valid identifier
start character.
Sourcepub fn peek(&mut self) -> Result<Token<'a>, ParseError>
pub fn peek(&mut self) -> Result<Token<'a>, ParseError>
Look at the next token without consuming it.
The result is cached: repeated peek() calls do not re-parse.
§Errors
Any tokenization error at the upcoming position.
Sourcepub fn read(&mut self) -> Result<Token<'a>, ParseError>
pub fn read(&mut self) -> Result<Token<'a>, ParseError>
Sourcepub fn line_col(&self, pos: usize) -> (u32, u32)
pub fn line_col(&self, pos: usize) -> (u32, u32)
Convert a byte offset into the input into a 1-based (line, column).
Column counts Unicode scalar values, not bytes. A pos past the end
of the input gives the position just after the last character. An
offset inside a UTF-8 character is treated as that character’s start.
Line and column use u32: inputs larger than ~4 GiB would wrap, but
that is not a realistic textproto size and this is error-reporting only.