Enum TokenKind 

pub enum TokenKind {
    Int(i64),
    Float(f64),
    Str(String),
    InterpolatedStr {
        parts: Vec<String>,
        exprs: Vec<String>,
    },
    Sym(String),
    True,
    False,
    Nil,
    Ident(String),
    LParen,
    RParen,
    LBracket,
    RBracket,
    LBrace,
    RBrace,
    Comma,
    Dot,
    Colon,
    PathSep,
    Label(String),
    Rocket,
    Pipe,
    Op(String),
    Doc {
        module: bool,
        text: String,
    },
    Comment(String),
    Newline,
    Eof,
}

Variants

Int(i64)

Float(f64)

Str(String)

InterpolatedStr

An interpolated string: alternating literal and expression parts.

"a#{x}b" lexes to ["a", "b"] literals with ["x"] between them — the expression is kept as SOURCE TEXT and parsed by the parser, which already knows how to parse an expression. Re-implementing expression lexing inside the string lexer would be a second parser, and the two would drift.

Fields

parts: Vec<String>

parts.len() == exprs.len() + 1, always — the literal before each expression, plus the tail. An empty literal is kept rather than dropped so that invariant holds for "#{a}#{b}" too.

exprs: Vec<String>

Sym(String)

:name — a Ruby symbol, which lowers to a tatara-lisp keyword.

True

False

Nil

Ident(String)

An identifier, or a keyword-like head (if, do, end, …). The parser decides which; the lexer does not need to know.

LParen

RParen

LBracket

RBracket

LBrace

RBrace

Comma

Dot

Colon

: in a hash literal (foo: 1) is folded into Label; a bare colon is retained for anything else.

PathSep

:: — the qualifier in retsu::first, package on the left and one of its definitions on the right. One token, so a qualified name never depends on how two colons happened to lex.

Label(String)

foo: — a hash-literal label. Lexing this as one token is what makes {foo: 1} and {:foo => 1} distinguishable at the parser without lookahead games.

Rocket

=> — the “rocket”.

Pipe

|> — the pipeline operator.

Op(String)

Any operator run: + - * / == != < <= > >= && || = ! %.

Doc

@doc """ … """ or @moduledoc """ … """: documentation text, its indentation and trailing whitespace stripped at lex time (crate::doc). Not trivia: the parser puts it in the tree.

Fields

module: bool

text: String

Comment(String)

Newline

Eof

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenKind

A token as an error message names it: its blue spelling in backticks, or, for the tokens that have no spelling, what it is in words.

This is what a parse error says it found. Errors used to format the token with {:?}, so an author who wrote [1, 2 y] read found Ident("y") — the name of a Rust enum variant, in a language whose surface has no such thing. A message is read by the person who wrote the source, so it speaks the source’s vocabulary.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind