Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub flags: TokenFlags,
    pub value: u32,
    pub span: Span,
}

One token, as the parser reads it.

Sixteen bytes, checked by a test below, and laid out the way PpToken is for the same reason.

Fields

kind: TokenKind

What it is.

flags: TokenFlags

Whether it started a line and whether anything came before it, carried through from the pp-token so that a diagnostic can say “did you mean to write this on its own line”.

value: u32

The symbol for an identifier, the index into the matching vector of Tokens for a constant or a literal, and zero for everything else.

span: Span

Where it came from, which for a run of string literals covers the whole run.

Implementations

impl Token

pub const fn is_eof(self) -> bool

Whether this is the end of the translation unit.

pub const fn keyword(self) -> Option<Keyword>

The keyword, when this is one.

pub const fn punct(self) -> Option<Punct>

The punctuator, when this is one.

pub const fn ident(self) -> Option<Symbol>

The identifier’s symbol, when this is one.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Token

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token