Enum lexer::token::TokenKind

pub enum TokenKind {
    NewLine,
    Illegal(String),
    UnterminatedComment,
    UnterminatedString,
    Eof,
    Identifier {
        name: String,
    },
    Int(String),
    Float(String),
    String(String),
    Assign,
    Plus,
    Minus,
    Mult,
    Div,
    Mod,
    Lt,
    LtEq,
    Gt,
    GtEq,
    Eq,
    NotEq,
    Comma,
    Semicolon,
    Colon,
    LParen,
    RParen,
    LCurly,
    RCurly,
    LBracket,
    RBracket,
    DefineFunction,
    Set,
    True,
    False,
    If,
    Otherwise,
    Return,
    Then,
    End,
    Repeat,
    Times,
    Until,
    Forever,
    Display,
    Not,
    And,
    Or,
}

Represents a specific variant of a token.

A TokenKind can be one of several variants, such as Identifier, Int, etc. Variants may contain additional data specific to that kind of token.

Variants

NewLine

Illegal(String)

Represents a character that does not match any other token.

UnterminatedComment

Represents a multi-line comment /* that does not have a corresponding */.

UnterminatedString

Represents a string literal that does not have a closing quote.

Eof

Represents the end of the file.

Identifier

Fields

name: String

Int(String)

Float(String)

String(String)

Assign

Plus

Minus

Mult

Div

Mod

Lt

LtEq

Gt

GtEq

Eq

NotEq

Comma

Semicolon

Colon

LParen

RParen

LCurly

RCurly

LBracket

RBracket

DefineFunction

Set

True

False

If

Otherwise

Return

Then

End

Repeat

Times

Until

Forever

Display

Not

And

Or

Implementations

impl TokenKind

pub fn lookup_ident(ident: &str) -> Self

Looks up an identifier and returns the corresponding token kind.

This function serves as a mapping, providing a single point of truth defining all the keywords in the language. The lexer module uses this function to generate tokens for keywords.

Arguments

• ident - The identifier to look up.

Returns

The corresponding token kind for the identifier.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenKind

Implements the Display trait for TokenKind. This allows TokenKind to be formatted as a string when using the write! macro.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenKind

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for TokenKind