Struct TokenRef 

pub struct TokenRef<'src> {
    pub kind: TokenKind,
    pub text: &'src str,
    pub start: usize,
    pub end: usize,
}

Borrowed view over token data for allocation-sensitive paths.

Unlike Token, this type borrows source text and does not allocate. Convert to Token explicitly with TokenRef::to_owned_token or From.

Fields

kind: TokenKind

Token classification for parser decision making

text: &'src str

Borrowed source text slice

start: usize

Starting byte position for error reporting and location tracking

end: usize

Ending byte position for span calculation and navigation

Implementations

impl<'src> TokenRef<'src>

pub fn new(kind: TokenKind, text: &'src str, start: usize, end: usize) -> Self

Create a borrowed token view with the given kind, source text, and byte span.

pub fn try_new( kind: TokenKind, text: &'src str, start: usize, end: usize, ) -> Result<Self, TokenSpanError>

Create a borrowed token view with checked span ordering.

Unlike TokenRef::new, this rejects spans where end < start.

pub fn new_checked( kind: TokenKind, text: &'src str, start: usize, end: usize, ) -> Result<Self, TokenSpanError>

Create a borrowed token view while enforcing span invariants.

Rules:

• start <= end
• zero-length spans are accepted for EOF and explicit synthetic unknown tokens

pub fn len(self) -> usize

Return the token span length in bytes.

pub fn is_empty(self) -> bool

Return whether the token span is empty.

pub fn span(self) -> (usize, usize)

Return the token span as (start, end).

pub fn display_name(self) -> &'static str

Return a human-readable display name for this token.

pub fn to_owned_token(self) -> Token

Convert this borrowed token view into an owned Token.

Trait Implementations

impl<'src> Clone for TokenRef<'src>

fn clone(&self) -> TokenRef<'src>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'src> Debug for TokenRef<'src>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<TokenRef<'_>> for Token

fn from(value: TokenRef<'_>) -> Self

Converts to this type from the input type.

impl<'src> PartialEq for TokenRef<'src>

fn eq(&self, other: &TokenRef<'src>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'src> Copy for TokenRef<'src>

impl<'src> Eq for TokenRef<'src>

impl<'src> StructuralPartialEq for TokenRef<'src>