Struct Lexer

Source

pub struct Lexer<'a> { /* private fields */ }

Expand description

A cursor over the bytes of a PDF file.

Offsets are relative to whatever slice the lexer was built over. The document layer passes the file from its header onwards, so a lexer position and a cross-reference offset mean the same thing.

use pdfrum_common::Limits;
use pdfrum_parser::{Lexer, Token};

let limits = Limits::default();
let mut lx = Lexer::new(b"12 0 obj % a comment\n<< /Type /Page >>");
assert_eq!(lx.next_word(&limits), Token::Number(b"12"));
assert_eq!(lx.next_word(&limits), Token::Number(b"0"));
assert_eq!(lx.next_word(&limits), Token::Keyword(b"obj"));
// Comments are invisible everywhere except inside strings.
assert!(matches!(lx.next_word(&limits), Token::Delim(_)));
assert_eq!(lx.next_word(&limits), Token::Name(b"Type"));

Implementations§

Source §

impl<'a> Lexer<'a>

Source

pub fn new(bytes: &'a [u8]) -> Self

A lexer positioned at the start of bytes.

Source

pub fn at(bytes: &'a [u8], pos: usize) -> Self

A lexer positioned at pos, clamped to the end of the input.

Source

pub fn bytes(&self) -> &'a [u8] ⓘ

The bytes being read.

Source

pub fn pos(&self) -> usize

The current offset.

Source

pub fn seek(&mut self, pos: usize)

Move to pos, clamped to the end of the input.

Source

pub fn at_eof(&self) -> bool

Whether the cursor is at or past the end.

Source

pub fn peek_byte(&self) -> Option<u8>

The byte at the cursor, without advancing.

Source

pub fn skip_to_word(&mut self)

Skip whitespace and comments, leaving the cursor on the first byte of the next token (or at the end).

A % runs to the next line ending, and the skipping repeats — so a block of comment lines costs one call.

Source

pub fn to_next_line(&mut self)

Move past the next line ending, so the cursor sits on the first byte of the following line.

A \r\n pair counts as one ending. This is how stream data finds its first byte after the stream keyword (ISO 32000-1 §7.3.8.1).

Source

pub fn skip_eol_marker(&mut self) -> usize

Consume one end-of-line marker if the cursor is on one, and report how many bytes it took: two for \r\n, one for a lone \r or \n, zero for anything else.

Source

pub fn next_word(&mut self, limits: &Limits) -> Token<'a>

Read the next token, skipping whitespace and comments first.

Words longer than limits.max_word_len are truncated in the returned token but consumed whole, so the cursor always lands past the run.

Source

pub fn peek_word(&mut self, limits: &Limits) -> Token<'a>

Read the next token and restore the cursor, so a caller can decide what to do without committing.

Source

pub fn read_literal_string(&mut self) -> Cow<'a, [u8]>

Read the body of a literal string, the ( already consumed (ISO 32000-1 §7.3.4.2).

Nested parentheses are kept as content and only an unescaped ) at depth zero ends the string. An end of file ends it too, silently, with whatever was read — the recovery scan depends on that, because it uses this function to skip over string bodies that may well be truncated.

The result borrows the file when no escape sequence forced a rewrite.

Source

pub fn read_hex_string(&mut self) -> Vec<u8> ⓘ

Read the body of a hexadecimal string, the < already consumed (ISO 32000-1 §7.3.4.3).

Every byte that is neither a hex digit nor > is skipped without comment — whitespace, NULs, letters, anything. A > or the end of file ends the string, and a dangling half byte is padded with a zero nibble, so <1A2 reads as 1A 20.

Source

pub fn search_back(&mut self, word: &[u8], window: usize) -> bool

Search backwards from the cursor for word as a whole word, within window bytes, and leave the cursor on its first byte.

“Whole word” means the neighbouring bytes are not regular or numeric; a delimiter beside the word is an acceptable boundary. The cursor’s own byte is inside the search, so a match may end at pos() rather than before it. This is how startxref is found in a file whose tail is otherwise junk.