Enum TokenKind 

pub enum TokenKind {
    Int(i64),
    Float(f64),
    Str(String),
    InterpolatedStr {
        parts: Vec<String>,
        exprs: Vec<String>,
    },
    Sym(String),
    True,
    False,
    Nil,
    Ident(String),
    LParen,
    RParen,
    LBracket,
    RBracket,
    LBrace,
    RBrace,
    Comma,
    Dot,
    Colon,
    Label(String),
    Rocket,
    Pipe,
    Op(String),
    Comment(String),
    Newline,
    Eof,
}

Variants

Int(i64)

Float(f64)

Str(String)

InterpolatedStr

An interpolated string: alternating literal and expression parts.

"a#{x}b" lexes to ["a", "b"] literals with ["x"] between them — the expression is kept as SOURCE TEXT and parsed by the parser, which already knows how to parse an expression. Re-implementing expression lexing inside the string lexer would be a second parser, and the two would drift.

Fields

parts: Vec<String>

parts.len() == exprs.len() + 1, always — the literal before each expression, plus the tail. An empty literal is kept rather than dropped so that invariant holds for "#{a}#{b}" too.

exprs: Vec<String>

Sym(String)

:name — a Ruby symbol, which lowers to a tatara-lisp keyword.

True

False

Nil

Ident(String)

An identifier, or a keyword-like head (if, do, end, …). The parser decides which; the lexer does not need to know.

LParen

RParen

LBracket

RBracket

LBrace

RBrace

Comma

Dot

Colon

: in a hash literal (foo: 1) is folded into Label; a bare colon is retained for anything else.

Label(String)

foo: — a hash-literal label. Lexing this as one token is what makes {foo: 1} and {:foo => 1} distinguishable at the parser without lookahead games.

Rocket

=> — the “rocket”.

Pipe

|> — the pipeline operator.

Op(String)

Any operator run: + - * / == != < <= > >= && || = ! %.

Comment(String)

Newline

Eof

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind