Enum Token 

pub enum Token {
    Start {
        kind: u32,
        pos: u32,
    },
    End {
        kind: u32,
        pos: u32,
        start_idx: u32,
    },
    Leaf {
        kind: u32,
        start: u32,
        end: u32,
    },
}

One parse-tree token. Internal nodes are bracketed by a matching Start/End pair; terminal/metasymbol matches are stored as Leaf.

kind is grammar-specific: each parser assigns its own integer ids to variables and terminal categories.

Variants

Start

Opening marker of a successful rule match.

pos is the input position where matching started. The matching End’s index is not stored on the Start — readers walk forward and use the start_idx back-pointer on End to associate the pair.

Fields

kind: u32

pos: u32

End

Closing marker of a successful rule match.

pos is the end input position (exclusive). start_idx is the index of the matching Start in ParserState::tokens.

Fields

kind: u32

pos: u32

start_idx: u32

Leaf

A terminal or metasymbol match.

Fields

kind: u32

start: u32

end: u32

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Token

impl Eq for Token

impl StructuralPartialEq for Token