Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub start: u32,
    pub end: u32,
    pub mate_plus_one: Option<NonZeroU32>,
}

Fields

kind: TokenKind

The token’s typed class.

start: u32

Inclusive start byte offset into the input.

end: u32

Exclusive end byte offset into the input.

mate_plus_one: Option<NonZeroU32>

The mate’s index plus one, or None. Read it through Token::mate and write it through Token::set_mate, which hold the encoding in one place; the field is named for what it stores so that reading it as an index does not compile.

Implementations

impl Token

pub fn new(kind: TokenKind, start: usize, end: usize) -> Self

Construct a token with no bracket mate.

Panics

When an offset does not fit the stored width, which needs an input over four gigabytes. Truncating instead would point a token at the wrong bytes.

pub fn span(&self) -> Range<usize>

The token’s byte span, for indexing the input it was lexed from.

pub fn start(&self) -> usize

The start offset as a usize, for arithmetic and indexing against everything else, which counts bytes in the machine’s width. The field is the storage and this is the view of it; the two never disagree, and a site that needs the wider one fails to compile rather than silently taking the narrower.

pub fn end(&self) -> usize

The end offset as a usize. See Token::start.

pub fn shift(&mut self, by: usize)

Move the span by by bytes, for a token lexed from a chunk and rebased onto the whole input.

Panics

When the shifted offset does not fit the stored width.

pub fn mate(&self) -> Option<usize>

The index of this token’s matching bracket, or None when it has none.

pub fn set_mate(&mut self, index: Option<usize>)

Record this token’s matching bracket, or clear it.

Panics

When the index does not fit the stored width. A stream that long would need more than four billion tokens, which no input this lexer can hold produces, and silently storing a wrapped index would pair the wrong brackets.

pub fn len(&self) -> usize

The byte length of the token’s span.

pub fn is_empty(&self) -> bool

Whether the token’s span is empty.

pub fn is_significant(&self) -> bool

Whether this token participates in token-mode matching (everything except insignificant whitespace).

Trait Implementations

impl Clone for Token

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Token

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token