Struct Token

pub struct Token {
    pub lexeme: String,
    pub tok_type: TokType,
    pub context: Option<TokenContext>,
}

Represents a source token

Unlike the spans found in crate::tokenize::Tokens, these tokens are of an owned type containing an owned String of the lexeme from source.

The typical way to obtain a Token is from the source_to_tokens function. But tokens can be created without source documents, too.

Fields

lexeme: String

The contents of the token, exactly as it appears in source

tok_type: TokType

the type of the token

context: Option<TokenContext>

Contextual information about the Token’s position in the source document (if available) Because tokens can be created without a source doc, this field is an Option and may be None.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token