Enum tremor_script::lexer::Token

pub enum Token<'input> {
    Whitespace(&'input str),
    NewLine,
    SingleLineComment(&'input str),
    ModComment(&'input str),
    DocComment(&'input str),
    Bad(String),
    Ident(Cow<'input, str>, bool),
    Dollar,
    Dot,
    Nil,
    BoolLiteral(bool),
    IntLiteral(i64),
    FloatLiteral(f64, String),
    TestLiteral(usize, Vec<String>),
    HereDoc(usize, Vec<String>),
    DQuote,
    StringLiteral(Cow<'input, str>),
    Let,
    Const,
    Match,
    Case,
    Of,
    When,
    End,
    Patch,
    Insert,
    Upsert,
    Update,
    Erase,
    Move,
    Copy,
    Merge,
    Drop,
    Default,
    Emit,
    For,
    Event,
    State,
    Present,
    Absent,
    Fun,
    Intrinsic,
    Module,
    DontCare,
    Recur,
    BSlash,
    Comma,
    Not,
    BitNot,
    And,
    Or,
    Xor,
    BitAnd,
    BitOr,
    BitXor,
    Eq,
    EqEq,
    NotEq,
    TildeEq,
    Tilde,
    Gte,
    Gt,
    Lte,
    Lt,
    RBitShiftSigned,
    RBitShiftUnsigned,
    LBitShift,
    Add,
    Sub,
    Mul,
    Div,
    Mod,
    Colon,
    ColonColon,
    EqArrow,
    Semi,
    LPatParen,
    LParen,
    RParen,
    LBrace,
    LPatBrace,
    RBrace,
    LBracket,
    LPatBracket,
    RBracket,
    EndOfStream,
    Select,
    From,
    Where,
    With,
    Order,
    Group,
    By,
    Having,
    Into,
    Create,
    Tumbling,
    Sliding,
    Window,
    Stream,
    Operator,
    Script,
    Set,
    Each,
    Define,
    Args,
    Use,
    As,
    LineDirective(Location, Cow<'input, str>),
    ConfigDirective,
}

A token in the source ( file, byte stream ), to be emitted by the Lexer The LALRPOP grammar uses these tokens and this custom lexer as it does not have a facility to ignore special tokens, to easily disambiguate tokens ( eg: ':' vs '::' vs ':=' ), nor to inject magic tokens ( bad token ) or support lexical streams ( different token streams drive by users, eg: emit ignorable tokens when syntax highlighting or error highlighting

Variants

Whitespace(&'input str)

Ignorable when parsing, significant when highlighting

NewLine

a new line

SingleLineComment(&'input str)

a singe line comment

ModComment(&'input str)

a mod comment

DocComment(&'input str)

a doc comment

Bad(String)

a BAD TOKEN

Ident(Cow<'input, str>, bool)

an ident

Dollar

the $ sign

Dot

a .

Nil

null

BoolLiteral(bool)

a boolean

IntLiteral(i64)

an integer

FloatLiteral(f64, String)

an float

TestLiteral(usize, Vec<String>)

a test literal

HereDoc(usize, Vec<String>)

an heredoc

DQuote

a double quote "

StringLiteral(Cow<'input, str>)

a string literal

Let

the let keywrod

Const

the const keyword

Match

the match keyword

Case

the case keyword

Of

the of keyword

When

the when keyword

End

the end keyword

Patch

the patch keyword

Insert

the insert keyword

Upsert

the upsert keyword

Update

the update keyword

Erase

the erase keyword

Move

the move keyword

Copy

the copy keyword

Merge

the merge keyword

Drop

the drop keyword

Default

the default keyword

Emit

the emit keyword

For

the for keyword

Event

the event keyword

State

the state keyword

Present

the present keyword

Absent

the absent keyword

Fun

the fun keyword

Intrinsic

the intrinsic keyword

Module

the mod keyword

DontCare

the _ token

Recur

the recure token

BSlash

the \ backslash

Comma

the , comma

Not

the not keyword

BitNot

bitwise not !

And

the and keyword

Or

the or keyword

Xor

the xor keyword

BitAnd

bitwise and &

BitOr

bitwise or |

BitXor

bitwise xor ^

Eq

equal =

EqEq

double equal ==

NotEq

not equal !=

TildeEq

tilde equal ~=

Tilde

tilde ~

Gte

greater than equal >=

Gt

grater than >

Lte

lower than equal <=

Lt

lower then <

RBitShiftSigned

Right bit shift signed >>

RBitShiftUnsigned

Right bit shift unsigned >>>

LBitShift

Left bit shift <<

Add

Add +

Sub

Substract -

Mul

Multiply *

Div

Divide /

Mod

Moduly %

Colon

Colon :

ColonColon

Double coilon ::

EqArrow

Effector array =>

Semi

Semicolon ;

LPatParen

Left Paren %(

LParen

Left Paren (

RParen

Right Paren )

LBrace

Left brace {

LPatBrace

Left pattern Bracket %{

RBrace

Right Brace }

LBracket

Left Bracket [

LPatBracket

Left pattern Bracket %[

RBracket

Right bracket ]

EndOfStream

End of stream token

Select

The select keyword

From

The from keyword

Where

The where keyword

With

The with keyword

Order

The order keyword

Group

the group keyword

By

The by keyword

Having

The having keyword

Into

The into keyword

Create

The create keyword

Tumbling

The tumbling keyword

Sliding

The sliding keyword

Window

The window keyword

Stream

The stream keyword

Operator

The operator keyword

Script

The script keyword

Set

The set keyword

Each

The each keyword

Define

The define keyword

Args

The args keyword

Use

The use keyword

As

The as keyword

LineDirective(Location, Cow<'input, str>)

Preprocessor directives

ConfigDirective

Config Directive

Methods

impl<'input> Token<'input>

pub fn prettify(&self) -> String

a prettified version of the token

Trait Implementations

impl<'input> Clone for Token<'input>

fn clone(&self) -> Token<'input>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'input> Debug for Token<'input>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<'input> Display for Token<'input>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<'input> PartialEq<Token<'input>> for Token<'input>

fn eq(&self, other: &Token<'input>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Token<'input>) -> bool

This method tests for !=.

impl<'input> StructuralPartialEq for Token<'input>