Enum Token 

pub enum Token<'a> {
    Number(&'a [u8]),
    Name(&'a [u8]),
    Keyword(&'a [u8]),
    Delim(Delim),
    Eof,
}

One syntactic token, borrowing from the file.

The tokenizer is deliberately shallow: it reports what shape a run of bytes has, never what it means. Whether 12 is a length, an object number, or an array element is the grammar’s business, so a number arrives as Token::Number carrying its spelling and the value parse happens one layer up.

Variants

Number(&'a [u8])

A word whose every byte is in the numeric class. Spelling is kept because --37, 1.2.3 and +-. all reach here, and only the value parse decides what they are worth.

Name(&'a [u8])

A name, without its leading slash and before escape decoding. An empty slice is the valid empty name /.

Keyword(&'a [u8])

A keyword or any other run of regular bytes: obj, endstream, true, or junk.

Delim(Delim)

A punctuation token: one delimiter, or the paired << and >>.

Eof

The file ended before another token began.

Implementations

impl<'a> Token<'a>

pub fn bytes(&self) -> &'a [u8]

The token’s bytes as the tokenizer stored them, for the callers that compare against literals. Punctuation answers with its spelling.

pub fn is_number(&self) -> bool

Whether this is a number-shaped word.

pub fn is_eof(&self) -> bool

Whether the file ended.

pub fn is_keyword(&self, word: &[u8]) -> bool

Whether this is the given keyword.

Trait Implementations

impl<'a> Clone for Token<'a>

fn clone(&self) -> Token<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Token<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Eq for Token<'a>

impl<'a> PartialEq for Token<'a>

fn eq(&self, other: &Token<'a>) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl<'a> StructuralPartialEq for Token<'a>