Enum TokenKind 

pub enum TokenKind {
    Eof,
    Identifier,
    QuotedIdentifier,
    Verbatim,
    String,
    Number,
    Comment,
    Operator,
    Dot,
    OpenParen,
    CloseParen,
    OpenBracket,
    CloseBracket,
    OpenBrace,
    CloseBrace,
    Comma,
    Semicolon,
    Unknown,
}

What a Token is.

Variants

Eof

End of input. Always the last token; text is empty.

Identifier

Source, let, Table.SelectRows, #table, Server.Name — keywords, dotted names, and the #-prefixed keyword family are all identifiers here; whether a word means a keyword or a name is resolution’s business, not the lexer’s.

QuotedIdentifier

#"1998 Sales" — a quoted identifier, "" as the escape.

Verbatim

#!"not parsed" — a verbatim literal (evaluates to an error value).

String

"text" — a text literal, "" as the escape.

Number

1, 1.5, 1.5E-10, 0xff.

Comment

// … or /* … */ — never produces a reference.

Operator

= < > <= >= <> + - * / & @ ! ? ?? => .. ...

Dot

. on its own — a dot that did not absorb into an identifier.

OpenParen

(

CloseParen

)

OpenBracket

[ — opens a field access ([Name]) or a record literal ([K = V]); the extractor tells the two apart.

CloseBracket

]

OpenBrace

{

CloseBrace

}

Comma

,

Semicolon

;

Unknown

Any other character. Never fails the scan.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenKind

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl Hash for TokenKind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind