Struct Token

pub struct Token {
    pub start: Position,
    pub leading_trivia: Vec<Trivia>,
    pub token_type: TokenType,
    pub trailing_trivia: Vec<Trivia>,
    pub end: Position,
}

A single token. Every lexable item becomes a token in Lexer::next_token().

Fields

start: Position

The starting position of this token

leading_trivia: Vec<Trivia>

The trivia before the token.

token_type: TokenType

The actual info of the token.

trailing_trivia: Vec<Trivia>

The trivia after the token.

end: Position

The ending position of this token.

Implementations

impl Token

pub const fn empty(token_type: TokenType) -> Token

Creates an empty token with the specified type. This is only used when creating tokens that don’t have actual positions.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl GetRange for Token

fn get_range(&self) -> Result<Range, GetRangeError>

Get the range of the node.

impl Hash for Token

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for Token

fn cmp(&self, other: &Token) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq<TokenType> for Token

fn eq(&self, other: &TokenType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Token

fn partial_cmp(&self, other: &Token) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Print for Token

fn print_final_trivia(&self) -> String

Prints only the very final trivia. Used for the default implementation of Print::print, which just joins Print::print_without_final_trivia and this function.

fn print_without_final_trivia(&self) -> String

Prints the whole token including all surrounding trivia, excluding the very last trailing trivia.

fn print(&self) -> String

Prints the whole token including all surrounding trivia.

impl Eq for Token

impl StructuralPartialEq for Token