Struct Token

pub struct Token<'text> {
    pub text: &'text str,
    pub token_type: TokenType,
    pub location: SourceLocation,
}

A token from a body of text.

The lifetime parameter 'text refers to the lifetime of the body of text that was tokenized, creating this token.

Fields

text: &'text str

The text of the token.

token_type: TokenType

The type of the token (Whitespace or Word).

location: SourceLocation

The location of the token in the source body of text.

Implementations

impl<'text> Token<'text>

pub fn new( text: &'text str, token_type: TokenType, location: SourceLocation, ) -> Token<'_>

Construct a Token. The lifetime parameter 'text refers to the lifetime of the text being tokenized.

Trait Implementations

impl<'text> Clone for Token<'text>

fn clone(&self) -> Token<'text>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'text> Debug for Token<'text>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'text> PartialEq for Token<'text>

fn eq(&self, other: &Token<'text>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'text> Copy for Token<'text>

impl<'text> StructuralPartialEq for Token<'text>