Enum TokenKind 

pub enum TokenKind<'a> {
    KEYWORD(&'a str),
    DELIMITER(&'a str),
    PUNCTUATION(&'a str),
    OPERATOR(&'a str),
    COMMENT(&'a str),
    WHITESPACE(&'a str),
    LITERAL(&'a str),
    IDENTIFIER,
}

Premade Token kinds for semantic (examples are not mandatory):

TokenKind	Explanation	Examples
KEYWORD	Reserved words	if return ...
DELIMITER	Paired delimiter symbols	() [] {} ...
PUNCTUATION	Punctuation symbols	; . ...
OPERATOR	Symbols that operates on arguments	+ - = ...
COMMENT	Line or block comments	// /* ... */ ...
WHITESPACE	Non-printable characters	-
LITERAL	Numerical, logical, textual values	1 true "true" ...
IDENTIFIER	Names assigned in a program	x temp PRINT ...

These token kinds (except IDENTIFIER) should be constructed with a name that can be used to differentiate tokens with same kind.

Variants

KEYWORD(&'a str)

DELIMITER(&'a str)

PUNCTUATION(&'a str)

OPERATOR(&'a str)

COMMENT(&'a str)

WHITESPACE(&'a str)

LITERAL(&'a str)

IDENTIFIER

Trait Implementations

impl<'a> Clone for TokenKind<'a>

fn clone(&self) -> TokenKind<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for TokenKind<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> PartialEq for TokenKind<'a>

fn eq(&self, other: &TokenKind<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> PartialOrd for TokenKind<'a>

fn partial_cmp(&self, other: &TokenKind<'a>) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl<'a> StructuralPartialEq for TokenKind<'a>