Enum TokenKind 

#[non_exhaustive]
#[repr(u8)]
pub enum TokenKind {
    Whitespace = 0,
    Comment = 1,
    BareWord = 2,
    Number = 3,
    StringLiteral = 4,
    QuotedIdentifier = 5,
    OpeningRoundBracket = 6,
    ClosingRoundBracket = 7,
    OpeningSquareBracket = 8,
    ClosingSquareBracket = 9,
    OpeningCurlyBrace = 10,
    ClosingCurlyBrace = 11,
    Comma = 12,
    Semicolon = 13,
    VerticalDelimiter = 14,
    Dot = 15,
    Asterisk = 16,
    HereDoc = 17,
    DollarSign = 18,
    Plus = 19,
    Minus = 20,
    Slash = 21,
    Percent = 22,
    Arrow = 23,
    QuestionMark = 24,
    Colon = 25,
    Caret = 26,
    DoubleColon = 27,
    Equals = 28,
    NotEquals = 29,
    Less = 30,
    Greater = 31,
    LessOrEquals = 32,
    GreaterOrEquals = 33,
    Spaceship = 34,
    PipeMark = 35,
    PipeOperator = 36,
    Concatenation = 37,
    Tilde = 38,
    TildeAsterisk = 39,
    NotTilde = 40,
    NotTildeAsterisk = 41,
    At = 42,
    DoubleAt = 43,
    EndOfStream = 44,
    Error = 45,
    ErrorMultilineCommentIsNotClosed = 46,
    ErrorSingleQuoteIsNotClosed = 47,
    ErrorDoubleQuoteIsNotClosed = 48,
    ErrorBackQuoteIsNotClosed = 49,
    ErrorSingleExclamationMark = 50,
    ErrorSinglePipeMark = 51,
    ErrorWrongNumber = 52,
    ErrorMaxQuerySizeExceeded = 53,
}

The TokenType enum of ClickHouse’s src/Parsers/Lexer.h, in the same order.

The discriminants are positional and only valid together with the lexer compiled from the same revision, which is what this crate guarantees. Upstream may add variants in a patch release, hence non_exhaustive.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Whitespace = 0

Comment = 1

BareWord = 2

Either keyword (SELECT) or identifier (column)

Number = 3

Always non-negative. No leading plus. 123 or something like 123.456e12, 0x123p12

StringLiteral = 4

‘hello word’, ‘hello’‘word’, ‘hello'word\’

QuotedIdentifier = 5

“x”, x

OpeningRoundBracket = 6

ClosingRoundBracket = 7

OpeningSquareBracket = 8

ClosingSquareBracket = 9

OpeningCurlyBrace = 10

ClosingCurlyBrace = 11

Comma = 12

Semicolon = 13

VerticalDelimiter = 14

Vertical delimiter \G

Dot = 15

Compound identifiers, like a.b or tuple access operator a.1, (x, y).2.

Asterisk = 16

Need to be distinguished from floating point number with omitted integer part: .1 Could be used as multiplication operator or on it’s own: “SELECT *”

HereDoc = 17

DollarSign = 18

Plus = 19

Minus = 20

Slash = 21

Percent = 22

Arrow = 23

->. Should be distinguished from minus operator.

QuestionMark = 24

Colon = 25

Caret = 26

DoubleColon = 27

Equals = 28

NotEquals = 29

Less = 30

Greater = 31

LessOrEquals = 32

GreaterOrEquals = 33

Spaceship = 34

<=>. Used in MySQL for NULL-safe equality comparison.

PipeMark = 35

PipeOperator = 36

|>. Pipe operator: FROM t |> WHERE x |> SELECT y

Concatenation = 37

String concatenation operator: ||

Tilde = 38

~. Used in PostgreSQL as the regular expression match operator.

TildeAsterisk = 39

~*. Used in PostgreSQL as the case-insensitive regular expression match operator.

NotTilde = 40

!~. Used in PostgreSQL as the regular expression not-match operator.

NotTildeAsterisk = 41

!~*. Used in PostgreSQL as the case-insensitive regular expression not-match operator.

At = 42

@. Used for specifying user names and also for MySQL-style variables.

DoubleAt = 43

@@. Used for MySQL-style global variables.

EndOfStream = 44

Order is important. EndOfStream goes after all usual tokens, and special error tokens goes after EndOfStream.

Error = 45

Something unrecognized.

ErrorMultilineCommentIsNotClosed = 46

Something is wrong and we have more information.

ErrorSingleQuoteIsNotClosed = 47

ErrorDoubleQuoteIsNotClosed = 48

ErrorBackQuoteIsNotClosed = 49

ErrorSingleExclamationMark = 50

ErrorSinglePipeMark = 51

ErrorWrongNumber = 52

ErrorMaxQuerySizeExceeded = 53

Implementations

impl TokenKind

pub const ALL: &'static [TokenKind]

Every kind, in discriminant order.

pub fn from_raw(raw: u8) -> Option<TokenKind>

The kind with this discriminant, as returned by the C API.

pub fn name(self) -> &'static str

The upstream enum variant name, e.g. "BareWord".

impl TokenKind

pub fn is_significant(self) -> bool

Neither whitespace nor a comment.

pub fn is_error(self) -> bool

One of the Error* kinds, which end lexing.

pub fn is_end(self) -> bool

EndOfStream.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenKind

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl Hash for TokenKind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for TokenKind

fn cmp(&self, other: &Self) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

fn clamp_to<R>(self, range: R) -> Self

where Self: Sized, R: ClampBounds<Self>,

🔬This is a nightly-only experimental API. (clamp_to)

Restrict a value to a certain range.

impl PartialEq for TokenKind

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl PartialOrd for TokenKind

fn partial_cmp(&self, other: &Self) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl StructuralPartialEq for TokenKind