Struct Token

pub struct Token<'source> {
    pub span: Range<usize>,
    pub kind: TokenKind,
    pub lexeme: &'source str,
}

A token produced by the tokenizer.

Fields

span: Range<usize>

The region of the source code that this token originated from.

kind: TokenKind

The kind of token.

lexeme: &'source str

The raw lexeme that was parsed into this token.

Implementations

impl Token<'_>

pub fn is_ignore(&self) -> bool

Returns true if the token represents a token that should be ignored by the parser.

pub fn is_significant_whitespace(&self) -> bool

Returns true if the token represents significant whitespace, i.e., the token is a newline.

Trait Implementations

impl<'source> Clone for Token<'source>

fn clone(&self) -> Token<'source>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'source> Debug for Token<'source>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'source> PartialEq for Token<'source>

fn eq(&self, other: &Token<'source>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'source> StructuralPartialEq for Token<'source>