#[repr(C)]
pub struct Token { pub token_type: i32, pub token_value: Bytes, pub loc: Loc, pub lex_state_before: LexState, pub lex_state_after: LexState, }
Expand description

A token that is emitted by a lexer and consumed by a parser

Fields

token_type: i32

Numeric representation of the token type, e.g. 42 (for example) for tINTEGER

token_value: Bytes

Value of the token, e.g “42” for 42

loc: Loc

Location of the token

lex_state_before: LexState

Lex state before reading the token

lex_state_after: LexState

Lex state after reading the token

Implementations

Returns a byte array of the token value

Consumes a token and returns an owned byte array of the token value

Converts token value into &str

Converts token to a string, replaces unknown chars to U+FFFD

Converts token to a string

Consumes a token and converts it into a string

Returns name of the token

Trait Implementations

Returns a copy of the value. Read more
Performs copy-assignment from source. Read more
Formats the value using the given formatter. Read more
This method tests for self and other values to be equal, and is used by ==. Read more
This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The resulting type after obtaining ownership.
Creates owned data from borrowed data, usually by cloning. Read more
Uses borrowed data to replace owned data, usually by cloning. Read more
The type returned in the event of a conversion error.
Performs the conversion.
The type returned in the event of a conversion error.
Performs the conversion.