Struct Token 

pub struct Token {
    pub text: TokenText,
    pub loc: Option<SourceLocation>,
    pub noexpand: Option<bool>,
    pub treat_as_relax: Option<bool>,
}

Represents a single token in the lexing process of LaTeX/KaTeX mathematical expressions.

Fields

text: TokenText

The raw text content of the token as extracted from the input string.

This preserves the original characters without any processing or normalization. For commands, this includes the backslash and command name (e.g., “\alpha”).

loc: Option<SourceLocation>

Optional source location information for error reporting and debugging.

Provides context about where in the original input this token originated. Used to generate meaningful error messages when parsing fails.

See Also

• SourceLocation for location details

noexpand: Option<bool>

Flag indicating whether this token should not be expanded during macro processing.

When set to true, prevents the parser from applying macro expansions to this token. This is useful for tokens that should be treated literally, such as in verbatim environments.

treat_as_relax: Option<bool>

Flag indicating whether this token should be treated as a \relax command.

\relax is a LaTeX primitive that does nothing but can be used to prevent unwanted expansions. When this flag is set, the token behaves as if it were a \relax command.

Implementations

impl Token

pub fn new<T>(text: T, loc: Option<SourceLocation>) -> Self

where T: Into<TokenText>,

Creates a new token with the specified text and optional source location.

pub fn text(&self) -> &str

Returns the token text as a string slice.

pub fn set_text<T>(&mut self, text: T)

where T: Into<TokenText>,

Mutably sets the textual payload of the token.

pub fn range<T: Into<TokenText>>(self, end_token: Self, text: T) -> Option<Self>

Computes a new token that encompasses the range from this token to the end_token.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ErrorLocationProvider for Token

Implementation of ErrorLocationProvider for Token.

This implementation allows Token to be used as an error location provider in the KaTeX parsing pipeline. It simply returns the token’s location information for error reporting.

Cross-references

• Part of the error reporting system in ParseError(crate::types::ParseError).
• Used by parsers to provide location context for syntax errors.

fn loc(&self) -> Option<&SourceLocation>

Get the source location if available

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Token

impl StructuralPartialEq for Token