Struct Tokenizer 

pub struct Tokenizer<'a> { /* private fields */ }

PDF tokenizer: produces tokens from a byte slice.

Implementations

impl<'a> Tokenizer<'a>

pub fn new(source: &'a [u8]) -> Self

Create a new tokenizer starting at the given byte slice.

pub fn new_at(source: &'a [u8], pos: usize) -> Self

Create a new tokenizer starting at the given position.

pub fn position(&self) -> usize

Current byte position in the source.

pub fn set_position(&mut self, pos: usize)

Set the current position.

pub fn source(&self) -> &'a [u8]

Access the underlying source bytes.

pub fn skip_whitespace_and_comments(&mut self)

Skip whitespace and comments, returning true if any were skipped.

pub fn next_token(&mut self) -> Option<Result<Token, PdfError>>

Read the next token, or None at EOF.

pub fn peek_token(&mut self) -> Option<Result<Token, PdfError>>

Returns the next token without consuming it (peek without advancing).

Corresponds to CPDF_StreamParser::PeekNextWord() in PDFium.