Enum Token 

pub enum Token {
    Ident(Name),
    Lit(Literal),
    OpenDelim(DelimToken),
    CloseDelim(DelimToken),
    Keyword(Kw),
    Period,
    Comma,
    Colon,
    Semicolon,
    Apostrophe,
    Ampersand,
    Arrow,
    Condition,
    LtGt,
    VarAssign,
    Lshift,
    Rshift,
    Eq,
    Neq,
    Lt,
    Leq,
    Gt,
    Geq,
    MatchEq,
    MatchNeq,
    MatchLt,
    MatchLeq,
    MatchGt,
    MatchGeq,
    Add,
    Sub,
    Mul,
    Div,
    Pow,
    Pipe,
    Qmark,
    Eof,
}

A primary token as emitted by the lexer.

Variants

Ident(Name)

A basic or extended identifier.

Lit(Literal)

A literal.

OpenDelim(DelimToken)

An opening delimiter.

CloseDelim(DelimToken)

A closing delimiter.

Keyword(Kw)

A keyword.

Period

Comma

Colon

Semicolon

Apostrophe

Ampersand

Arrow

Condition

LtGt

VarAssign

Lshift

Rshift

Eq

Neq

Lt

Leq

Gt

Geq

MatchEq

MatchNeq

MatchLt

MatchLeq

MatchGt

MatchGeq

Add

Sub

Mul

Div

Pow

Pipe

Qmark

Eof

The end of the input file.

Implementations

impl Token

pub fn as_str(self) -> &'static str

pub fn is_ident(self) -> bool

Checks if this token is a identifier.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for Token

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T> TokenStream<Token> for BasicParser<T>

where T: Grinder<Item = Option<u8>, Error = DiagBuilder2>,

fn peek(&mut self, offset: usize) -> Spanned<Token>

Look ahead at a token in the stream.

fn bump(&mut self)

Consume the current token.

fn consumed(&self) -> usize

Get the number of tokens consumed. Excludes tokens skipped with skip.

fn last_span(&self) -> Span

Get the span of the last token consumed token (bumped or skipped).

fn emit(&mut self, diag: DiagBuilder2)

Emit a diagnostic.

fn severity(&self) -> Severity

Get the severity of the worst diagnostic emitted so far.

fn skip(&mut self)

Skip the current token. Usually the same as bump, but may be used to keep skipped tokens out of the consumed tokens count by some parsers.

fn last_loc(&self) -> Location

Get the tail location of the last consumed token (bumped or skipped).

fn is_fatal(&self) -> bool

Check whether a fatal diagnostic has been emitted.

fn is_error(&self) -> bool

Check whether an error diagnostic has been emitted.

impl Copy for Token

impl Eq for Token

impl StructuralPartialEq for Token