Struct Token 

pub struct Token {
    pub token_type: TokenType,
    pub literal: SmartString,
    pub position: Position,
    pub quoted: bool,
}

Token represents a lexical token

Uses SmartString for literal to avoid heap allocation for tokens up to 24 bytes (covers most SQL tokens: keywords, operators, short identifiers, numbers). For error tokens, the literal field contains the error message.

Fields

token_type: TokenType

The type of the token

literal: SmartString

The literal string value (SmartString inlines strings up to 24 bytes) For error tokens, this contains the error message instead.

position: Position

The position in the source

quoted: bool

Whether this token was double-quoted (identifier with string fallback)

Implementations

impl Token

pub fn new( token_type: TokenType, literal: impl AsRef<str>, position: Position, ) -> Self

Create a new token

pub fn new_quoted( token_type: TokenType, literal: impl AsRef<str>, position: Position, ) -> Self

Create a new double-quoted token (identifier with string fallback)

pub fn error( message: impl AsRef<str>, _literal: impl AsRef<str>, position: Position, ) -> Self

Create an error token (error message is stored in literal field)

pub fn eof(position: Position) -> Self

Create an EOF token

pub fn is_eof(&self) -> bool

Check if this is an EOF token

pub fn is_error(&self) -> bool

Check if this is an error token

pub fn is_keyword(&self, keyword: &str) -> bool

Check if this is a keyword with the given value (case-insensitive)

pub fn is_operator(&self, op: &str) -> bool

Check if this is an operator with the given value

pub fn is_punctuator(&self, punct: &str) -> bool

Check if this is a punctuator with the given value

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token