Struct lexer::token::Token

pub struct Token {
    pub kind: TokenKind,
    pub span: Span,
}

Represents a token in the lexer.

Tokens are the smallest units of a programming language. They represent individual elements such as keywords, identifiers, literals, and symbols. Tokens are used by the lexer to break down the source code into meaningful components that can be processed by the parser.

Tokens are useful because they provide a structured representation of the source code, allowing for easier analysis, interpretation, and transformation of the code. They serve as the foundation for building compilers, interpreters, and other language processing tools.

Fields

kind: TokenKind

span: Span

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

Implements the Display trait for the Token struct. This allows the Token struct to be formatted as a string.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token