Struct Token 

pub struct Token {
    pub token_type: TokenType,
    pub lexeme: String,
    pub line: usize,
    pub column: usize,
}

Represents a token with its type, value, and position in source

Fields

token_type: TokenType

lexeme: String

line: usize

column: usize

Implementations

impl Token

pub fn new( token_type: TokenType, lexeme: String, line: usize, column: usize, ) -> Self

Create a new token

pub fn location(&self) -> SourceLocation

Get the source location of this token

pub fn is_keyword(&self) -> bool

Check if token is a keyword

pub fn is_operator(&self) -> bool

Check if token is an operator

pub fn is_delimiter(&self) -> bool

Check if token is a delimiter

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token