Enum TokenKind 

pub enum TokenKind {
    Name,
    Keyword,
    Number,
    String,
    FString,
    TemplateString,
    Operator,
    Newline,
    Indent,
    Dedent,
    Trivia,
    End,
}

Kind of a lossless lexical token.

Variants

Name

Identifier, including contextual soft-keyword spellings.

Keyword

Reserved keyword.

Number

Numeric literal with its original spelling retained.

String

Ordinary string or bytes literal.

FString

Formatted string literal.

TemplateString

Python 3.14 template string literal.

Operator

Operator or delimiter.

Newline

Physical line ending.

Indent

Significant indentation increase.

Dedent

Significant indentation decrease (zero-width).

Trivia

Spaces, tabs, form feeds, comments, or escaped newlines.

End

End marker (zero-width).

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind