Struct Tokenizer 

pub struct Tokenizer<'a> { /* private fields */ }

Available on crate feature text only.

Stateful textproto tokenizer.

Holds a cursor into the input string and a small stack of open delimiters. peek caches its result so it is cheap to call repeatedly before read.

Implementations

impl<'a> Tokenizer<'a>

pub fn new(input: &'a str) -> Self

Create a tokenizer positioned at the start of input.

A leading UTF-8 byte-order mark (U+FEFF) is silently skipped — some editors prepend one when saving, and it is not a valid identifier start character.

pub fn peek(&mut self) -> Result<Token<'a>, ParseError>

Look at the next token without consuming it.

The result is cached: repeated peek() calls do not re-parse.

Errors

Any tokenization error at the upcoming position.

pub fn read(&mut self) -> Result<Token<'a>, ParseError>

Consume and return the next token.

Errors

Any tokenization error at the current position.

pub fn line_col(&self, pos: usize) -> (u32, u32)

Convert a byte offset into the input into a 1-based (line, column).

Column counts Unicode scalar values, not bytes. pos past the end of input clamps to the final position.

Line and column use u32: inputs larger than ~4 GiB would wrap, but that is not a realistic textproto size and this is error-reporting only.