Enum TokenKind

pub enum TokenKind {
    NewLine,
    Whitespace,
    Comment,
    Eq,
    NotEq,
    ApproxEq,
    ApproxNotEq,
    Add,
    Sub,
    Mul,
    Div,
    Mod,
    Exp,
    Greater,
    GreaterEq,
    Less,
    LessEq,
    Not,
    Factorial,
    And,
    Or,
    BitAnd,
    BitOr,
    BitNot,
    BitRight,
    BitLeft,
    Assign,
    AddAssign,
    SubAssign,
    MulAssign,
    DivAssign,
    ModAssign,
    ExpAssign,
    AndAssign,
    OrAssign,
    BitAndAssign,
    BitOrAssign,
    BitRightAssign,
    BitLeftAssign,
    Bin,
    Oct,
    Hex,
    Name,
    Keyword,
    Comma,
    OpenParen,
    CloseParen,
    OpenCurly,
    CloseCurly,
    OpenSquare,
    CloseSquare,
    Quote,
    Semicolon,
    Int,
    Boolean,
    Dot,
    RangeHalfOpen,
    RangeClosed,
    Symbol,
}

The different kinds of tokens that can be produced by the tokenizer.

Variants

NewLine

Whitespace

Comment

Eq

NotEq

ApproxEq

ApproxNotEq

Add

Sub

Mul

Div

Mod

Exp

Greater

GreaterEq

Less

LessEq

Not

Factorial

And

Or

BitAnd

BitOr

BitNot

BitRight

BitLeft

Assign

AddAssign

SubAssign

MulAssign

DivAssign

ModAssign

ExpAssign

AndAssign

OrAssign

BitAndAssign

BitOrAssign

BitRightAssign

BitLeftAssign

Bin

Oct

Hex

Name

Keyword

Comma

OpenParen

CloseParen

OpenCurly

CloseCurly

OpenSquare

CloseSquare

Quote

Semicolon

Int

Boolean

Dot

RangeHalfOpen

RangeClosed

Symbol

Implementations

impl TokenKind

pub fn is_ignore(self) -> bool

Returns true if the token represents a token that should be ignored by the parser.

pub fn is_significant_whitespace(self) -> bool

Returns true if the token represents significant whitespace.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'s> Logos<'s> for TokenKind

type Error = ()

Error type returned by the lexer. This can be set using #[logos(error = MyError)]. Defaults to () if not set.

type Extras = ()

Associated type Extras for the particular lexer. This can be set using #[logos(extras = MyExtras)] and accessed inside callbacks.

type Source = str

Source type this token can be lexed from. This will default to str, unless one of the defined patterns explicitly uses non-unicode byte values or byte slices, in which case that implementation will use [u8].

fn lex(lex: &mut Lexer<'s, Self>)

The heart of Logos. Called by the Lexer. The implementation for this function is generated by the logos-derive crate.

fn lexer(source: &'source Self::Source) -> Lexer<'source, Self>

where Self::Extras: Default,

Create a new instance of a Lexer that will produce tokens implementing this Logos.

fn lexer_with_extras( source: &'source Self::Source, extras: Self::Extras, ) -> Lexer<'source, Self>

Create a new instance of a Lexer with the provided Extras that will produce tokens implementing this Logos.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind