Struct Tokenizer 

pub struct Tokenizer<'a> { /* private fields */ }

PDF tokenizer operating on a byte slice.

Implementations

impl<'a> Tokenizer<'a>

pub fn new(data: &'a [u8]) -> Self

pub fn new_at(data: &'a [u8], pos: usize) -> Self

Create a tokenizer starting at a specific offset.

pub fn pos(&self) -> usize

Current byte position in the data.

pub fn set_pos(&mut self, pos: usize)

Set position directly.

pub fn peek_byte(&self) -> Option<u8>

Peek at the current byte without consuming it.

pub fn is_eof(&self) -> bool

Check if we’ve reached the end of data.

pub fn data(&self) -> &'a [u8]

Get a slice of the underlying data.

pub fn next_token(&mut self) -> Result<Option<Token>>

Read the next token, skipping whitespace and comments.

pub fn skip_whitespace_and_comments(&mut self)

Skip whitespace and comments.

pub fn skip_whitespace(&mut self)

Skip whitespace only (no comments).