Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub text: Arc<str>,
    pub start: usize,
    pub end: usize,
}

Token produced by the lexer and consumed by the parser.

Stores the token kind, original source text, and byte span. The text is kept in an Arc<str> so buffering and lookahead can clone tokens cheaply.

Fields

kind: TokenKind

Token classification for parser decision making

text: Arc<str>

Original source text for precise reconstruction

start: usize

Starting byte position for error reporting and location tracking

end: usize

Ending byte position for span calculation and navigation

Implementations

impl Token

pub fn new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Self

Create a new token with the given kind, source text, and byte span.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Sub, "sub", 0, 3);
assert_eq!(tok.kind, TokenKind::Sub);
assert_eq!(&*tok.text, "sub");

pub fn try_new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Result<Self, TokenSpanError>

Create a token with checked span ordering.

Unlike Token::new, this rejects spans where end < start.

pub fn new_checked( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Result<Self, TokenSpanError>

Create a token while enforcing span invariants.

Rules:

• start <= end
• zero-length spans are accepted for EOF and explicit synthetic unknown tokens

pub fn eof_at(pos: usize) -> Self

Create an EOF token at pos.

pub fn unknown_at(text: impl Into<Arc<str>>, start: usize, end: usize) -> Self

Create an unknown (synthetic) token at start..end.

pub fn span(&self) -> TokenSpan

Return this token’s byte span.

pub fn range(&self) -> Range<usize>

Return this token’s byte span as Range<usize>.

pub fn with_span( &self, start: usize, end: usize, ) -> Result<Self, TokenSpanError>

Clone this token with a new checked span.

pub fn with_kind(&self, kind: TokenKind) -> Self

Clone this token with a new token kind.

pub fn len(&self) -> usize

Return the token span length in bytes.

This uses saturating subtraction so malformed spans (where end < start) are treated as zero-length instead of underflowing.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Identifier, "foo", 10, 13);
assert_eq!(tok.len(), 3);

pub fn is_empty(&self) -> bool

Return whether the token span is empty.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Eof, "", 8, 8);
assert!(tok.is_empty());

pub fn display_name(&self) -> &'static str

Return a human-readable display name for this token.

pub fn as_ref_token(&self) -> TokenRef<'_>

Return a borrowed token view over this token.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<TokenRef<'_>> for Token

fn from(value: TokenRef<'_>) -> Self

Converts to this type from the input type.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token