Struct Lexeme

pub struct Lexeme<LexemeType> {
    pub lexeme_type: LexemeType,
    pub contents: String,
}

A lexeme extracted from input text by a lexical analyzers.

Lexemes, also known as “tokens”, are sequences of consecutive characters separated from input text, and classified into categories (such as keywords, identifiers, operators), during the lexical analysis phase of the syntax-parsing pipeline. They represent atomic units of syntactic meaning.

Fields

lexeme_type: LexemeType

The type (category) of the lexeme.

contents: String

The original text that constituted the lexeme.

Implementations

impl<LexemeType> Lexeme<LexemeType>

pub fn new(lexeme_type: LexemeType, contents: &str) -> Self

Creates a new Lexeme of the given lexeme_type with the given contents.

Trait Implementations

impl<LexemeType: Debug> Debug for Lexeme<LexemeType>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<LexemeType: PartialEq> PartialEq for Lexeme<LexemeType>

fn eq(&self, other: &Lexeme<LexemeType>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<LexemeType: Eq> Eq for Lexeme<LexemeType>

impl<LexemeType> StructuralPartialEq for Lexeme<LexemeType>