Struct Tok 

pub struct Tok {
    pub kind: PpTokenKind,
    pub flags: TokenFlags,
    pub value: Option<Symbol>,
    pub span: Span,
    pub expansion: Span,
    pub hides: HideSet,
    pub trace: TraceId,
    /* private fields */
}

A token in flight through macro expansion.

Construct one from a lexed token with Tok::new. The fields are readable because everything downstream matches on them, but the placemarker flag is not settable from outside the crate, because a placemarker escaping the expander would be a token with no spelling and no meaning.

Fields

kind: PpTokenKind

What kind of token this is.

flags: TokenFlags

Whitespace and origin flags, carried through from the lexer.

value: Option<Symbol>

The interned spelling, or None for a punctuator, whose spelling is fixed.

span: Span

Where the token is spelled: in a macro body for a token that came from one, in the user’s file for a token that did not.

expansion: Span

Where the outermost macro invocation that produced this token was written, or Span::DUMMY for a token the user wrote directly.

This is the span a diagnostic points at, with span becoming a note, which is what makes an error three macros deep readable.

hides: HideSet

The macro names that must not expand this token again.

trace: TraceId

The chain of macros this token came out of, innermost first, or TraceId::NONE for a token the user wrote.

span says where the text is and expansion says where the user was standing. This says how one became the other, which is the part a reader cannot reconstruct by hand once there is more than one macro involved.

Implementations

impl Tok

pub fn new(pp: PpToken) -> Tok

A token straight from the lexer, with an empty hide set and no expansion point.

pub fn punct(self) -> Option<Punct>

The punctuator this token is, if it is one.

pub fn is(self, p: Punct) -> bool

Whether this token is the punctuator p.

pub fn ident(self) -> Option<Symbol>

The identifier this token is, if it is one.

pub fn is_placemarker(self) -> bool

Whether this is the empty token ## uses for an absent argument.

pub fn report_span(self) -> Span

The span to report a diagnostic about this token at.

The invocation point when there is one, because a user reading an error wants the line they wrote, not a line in a header they have never opened.

Trait Implementations

impl Clone for Tok

fn clone(&self) -> Tok

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Tok

impl Debug for Tok

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Tok

impl PartialEq for Tok

fn eq(&self, other: &Tok) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Tok