Enum Token 

pub enum Token {
    Keyword {
        keyword: Keyword,
        original: String,
    },
    Identifier(String),
    DelimitedIdentifier(String),
    Number(String),
    String(String),
    BlobLiteral(Vec<u8>),
    Symbol(char),
    Operator(MultiCharOperator),
    SessionVariable(String),
    UserVariable(String),
    Placeholder,
    NumberedPlaceholder(usize),
    NamedPlaceholder(String),
    Semicolon,
    Comma,
    LParen,
    RParen,
    Eof,
}

SQL Token produced by the lexer.

Variants

Keyword

SQL keyword (SELECT, FROM, etc.) Stores both the keyword variant and original text for error messages. The original text preserves user’s input case (e.g., “SeLeCt”).

Fields

keyword: Keyword

original: String

Identifier(String)

Identifier (table name, column name, etc.)

DelimitedIdentifier(String)

Delimited identifier (“columnName” - case-sensitive, can use reserved words)

Number(String)

Numeric literal (42, 3.14, etc.)

String(String)

String literal (‘hello’)

BlobLiteral(Vec<u8>)

Blob literal (x’48454C4C4F’ or X’1234’)

Symbol(char)

Single character symbols (+, -, *, /, =, <, >, etc.)

Operator(MultiCharOperator)

Multi-character operators (<=, >=, !=, <>, ||)

SessionVariable(String)

Session variable (@@variable, @@session.variable, @@global.variable)

UserVariable(String)

User variable (@variable)

Placeholder

Parameter placeholder (?) for prepared statements The index is assigned during parsing (0-indexed, in order of appearance)

NumberedPlaceholder(usize)

Numbered parameter placeholder ($1, $2, etc.) for prepared statements PostgreSQL-style: 1-indexed as written in SQL ($1 = first parameter)

NamedPlaceholder(String)

Named parameter placeholder (:name) for prepared statements Used by many ORMs and applications for readability

Semicolon

Semicolon (statement terminator)

Comma

Comma (separator)

LParen

Left parenthesis

RParen

Right parenthesis

Eof

End of input

Implementations

impl Token

pub fn to_sql(&self) -> String

Convert token back to valid SQL string. This is the inverse of lexing - it produces SQL that can be re-parsed.

pub fn syntax_error(&self) -> String

Generate a SQLite-compatible syntax error message for this token.

SQLite uses the format: near "TOKEN": syntax error where TOKEN is the actual text that caused the error.

Special cases:

• EOF (end of input) returns “incomplete input” (SQLite convention for truncated statements)
• Keywords preserve original case from user input (e.g., “SeLeCt”)

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token