Enum TokenKind 

#[non_exhaustive]
pub enum TokenKind {
    LowerId {
        qualifier: Option<ModuleName>,
        name: Identifier,
    },
    UpperId {
        qualifier: Option<ModuleName>,
        name: Identifier,
    },
    Op(Operator),
    IntLit(String),
    DecimalLit(String),
    StringLit(String),
    CharLit(String),
    LParen,
    RParen,
    LBracket,
    RBracket,
    LBrace,
    RBrace,
    Comma,
    Semi,
    Backtick,
    VLBrace,
    VRBrace,
    VSemi,
}

Token category plus normalized token payload.

std::fmt::Display prints a normalized spelling for diagnostics, logs, and compact test output. It is not a lossless source renderer: trivia, original literal escapes, and the distinction between real and virtual layout punctuation are intentionally not preserved. Use render_lossless when source-exact reconstruction matters.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

LowerId

Lowercase-initial identifier, possibly qualified: foo, Map.lookup.

Fields

qualifier: Option<ModuleName>

name: Identifier

UpperId

Uppercase-initial identifier, possibly qualified: Foo, DA.Set.Set.

Fields

qualifier: Option<ModuleName>

name: Identifier

Op(Operator)

Symbolic operator: +, <-, ->, =, =>, ::, ., \, …

IntLit(String)

Integer literal payload as normalized token text.

DecimalLit(String)

Decimal literal payload as normalized token text.

StringLit(String)

String literal value after escape decoding. Use the token span to recover original quotes and escape spelling.

CharLit(String)

Character literal value after escape decoding. Use the token span to recover original quotes and escape spelling.

LParen

(.

RParen

).

LBracket

[.

RBracket

].

LBrace

Explicit { from source.

RBrace

Explicit } from source.

Comma

,.

Semi

Explicit ; from source.

Backtick

`.

VLBrace

Layout-inserted virtual open brace (block start).

VRBrace

Layout-inserted virtual close brace (block end).

VSemi

Layout-inserted virtual semicolon (new item at block indentation).

Implementations

impl TokenKind

pub const fn keyword(&self) -> Option<&str>

The identifier text if this is an unqualified lowercase identifier — how the parser checks for (contextual) keywords.

pub fn is_keyword(&self, kw: &str) -> bool

pub fn is_op(&self, op: &str) -> bool

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenKind