Enum Token 

pub enum Token {
    LParen,
    RParen,
    Quote,
    Quasiquote,
    Unquote,
    UnquoteSplice,
    Dot,
    True,
    False,
    VectorOpen,
    BytevectorOpen,
    SyntaxQuote,
    DatumComment,
    Number(isize),
    Float(f64),
    Symbol {
        len: usize,
    },
    Char(char),
    String {
        len: usize,
    },
}

A token produced by the lexer.

Tokens are lightweight, Copy, and do not allocate. Symbol and string content is accessed through the lexer’s buffers or the original input after a token is returned.

Note: Eq is intentionally not derived because Token::Float contains an fsize (floating-point) value, and NaN != NaN breaks Eq semantics.

Variants

LParen

(

RParen

)

Quote

' (quote shorthand)

Quasiquote

` (quasiquote shorthand)

Unquote

, (unquote shorthand)

UnquoteSplice

,@ (unquote-splicing shorthand)

Dot

. (dot for dotted pairs) — only emitted when dot is followed by whitespace or )

True

#t or #T

False

#f or #F

VectorOpen

#( (vector literal open)

BytevectorOpen

#u8( (bytevector literal open)

SyntaxQuote

#' (syntax quote)

DatumComment

#; (datum comment — parser skips next datum)

Number(isize)

Integer number literal (value already parsed)

Float(f64)

Floating-point number literal (value already parsed)

Symbol

Symbol — raw bytes are in input[start..start+len] (not yet lowercased). Use Lexer::symbol_bytes to get the lowercased bytes.

Fields

len: usize

Length of the symbol name in bytes

Char(char)

Character literal (#\a, #\space, #\newline, etc.)

String

String literal — processed characters are in the lexer’s string buffer. Use Lexer::string_chars to access the character data.

Escape sequences are already processed.

Fields

len: usize

Number of characters in the string

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Token

impl StructuralPartialEq for Token