Enum TokenKind 

#[non_exhaustive]
pub enum TokenKind {
    Whitespace,
    Newline,
    Indent,
    LineComment,
    BlockComment,
    String,
    Interpolation,
    Number,
    Identifier,
    Regex,
    Punctuation,
    Unterminated,
}

What a token is, at the lexical level only.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Whitespace

Spaces and tabs; newlines are their own kind.

Newline

Indent

Leading whitespace of a line in an indentation-sensitive language.

LineComment

BlockComment

String

String, character, template or raw-string literal, quotes included.

Interpolation

Interpolated section of a template literal, ${ and } included.

Number

Identifier

Regex

Regular-expression literal in languages that have them.

Punctuation

Unterminated

A block comment or string that the file ends inside.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenKind

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind