Enum RawTokenKind 

pub enum RawTokenKind {
    LineComment {
        is_doc: bool,
    },
    BlockComment {
        is_doc: bool,
        terminated: bool,
    },
    Whitespace,
    Ident,
    Literal {
        kind: RawLiteralKind,
    },
    Eq,
    Lt,
    Le,
    EqEq,
    Ne,
    Ge,
    Gt,
    AndAnd,
    OrOr,
    Not,
    Tilde,
    Walrus,
    PlusPlus,
    MinusMinus,
    StarStar,
    BinOp(BinOpToken),
    BinOpEq(BinOpToken),
    At,
    Dot,
    Comma,
    Semi,
    Colon,
    Arrow,
    FatArrow,
    Question,
    OpenDelim(Delimiter),
    CloseDelim(Delimiter),
    Unknown,
    Eof,
}

Common lexeme types.

Variants

LineComment

// comment

/// doc comment

Fields

is_doc: bool

BlockComment

/* block comment */

/** block doc comment */

Fields

is_doc: bool

terminated: bool

Whitespace

Any whitespace character sequence.

Ident

ident or continue

At this step, keywords are also considered identifiers.

Literal

Examples: 123, 0x123, hex"123". Note that _ is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See RawLiteralKind for more details.

Fields

kind: RawLiteralKind

Eq

=

Lt

<

Le

<=

EqEq

==

Ne

!=

Ge

>=

Gt

>

AndAnd

&&

OrOr

||

Not

!

Tilde

~

Walrus

:=

PlusPlus

++

MinusMinus

--

StarStar

**

BinOp(BinOpToken)

A binary operator token.

BinOpEq(BinOpToken)

A binary operator token, followed by an equals sign (=).

At

@

Dot

.

Comma

,

Semi

;

Colon

:

Arrow

->

FatArrow

=>

Question

?

OpenDelim(Delimiter)

An opening delimiter (e.g., {).

CloseDelim(Delimiter)

A closing delimiter (e.g., }).

Unknown

Unknown token, not expected by the lexer, e.g. №

Eof

End of input.

Implementations

impl RawTokenKind

pub const fn is_eof(&self) -> bool

Returns true if this token is EOF.

pub const fn is_comment(&self) -> bool

Returns true if this token is a line comment or a block comment.

pub const fn is_trivial(&self) -> bool

Returns true if this token is a whitespace, line comment, or block comment.

Trait Implementations

impl Clone for RawTokenKind

fn clone(&self) -> RawTokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RawTokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for RawTokenKind

fn eq(&self, other: &RawTokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for RawTokenKind

impl Eq for RawTokenKind

impl StructuralPartialEq for RawTokenKind

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 3 bytes

Size for each variant:

• LineComment: 2 bytes
• BlockComment: 3 bytes
• Whitespace: 0 bytes
• Ident: 0 bytes
• Literal: 3 bytes
• Eq: 0 bytes
• Lt: 0 bytes
• Le: 0 bytes
• EqEq: 0 bytes
• Ne: 0 bytes
• Ge: 0 bytes
• Gt: 0 bytes
• AndAnd: 0 bytes
• OrOr: 0 bytes
• Not: 0 bytes
• Tilde: 0 bytes
• Walrus: 0 bytes
• PlusPlus: 0 bytes
• MinusMinus: 0 bytes
• StarStar: 0 bytes
• BinOp: 2 bytes
• BinOpEq: 2 bytes
• At: 0 bytes
• Dot: 0 bytes
• Comma: 0 bytes
• Semi: 0 bytes
• Colon: 0 bytes
• Arrow: 0 bytes
• FatArrow: 0 bytes
• Question: 0 bytes
• OpenDelim: 2 bytes
• CloseDelim: 2 bytes
• Unknown: 0 bytes
• Eof: 0 bytes