Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub text: Arc<str>,
    pub start: usize,
    pub end: usize,
}

Token types and token stream for lexer output. Token produced by the lexer and consumed by the parser.

Stores the token kind, original source text, and byte span. The text is kept in an Arc<str> so buffering and lookahead can clone tokens cheaply.

Fields

kind: TokenKind

Token classification for parser decision making

text: Arc<str>

Original source text for precise reconstruction

start: usize

Starting byte position for error reporting and location tracking

end: usize

Ending byte position for span calculation and navigation

Implementations

impl Token

pub fn new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Token

Create a new token with the given kind, source text, and byte span.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Sub, "sub", 0, 3);
assert_eq!(tok.kind, TokenKind::Sub);
assert_eq!(&*tok.text, "sub");

pub fn try_new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Result<Token, TokenSpanError>

Create a token with checked span ordering.

Unlike Token::new, this rejects spans where end < start.

pub fn new_checked( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Result<Token, TokenSpanError>

Create a token while enforcing span invariants.

Rules:

• start <= end
• zero-length spans are accepted for EOF and explicit synthetic unknown tokens

pub fn eof_at(pos: usize) -> Token

Create an EOF token at pos.

pub fn unknown_at(text: impl Into<Arc<str>>, start: usize, end: usize) -> Token

Create an unknown (synthetic) token at start..end.

pub fn span(&self) -> TokenSpan

Return this token’s byte span.

pub fn range(&self) -> Range<usize>

Return this token’s byte span as Range<usize>.

pub fn with_span( &self, start: usize, end: usize, ) -> Result<Token, TokenSpanError>

Clone this token with a new checked span.

pub fn with_kind(&self, kind: TokenKind) -> Token

Clone this token with a new token kind.

pub fn len(&self) -> usize

Return the token span length in bytes.

This uses saturating subtraction so malformed spans (where end < start) are treated as zero-length instead of underflowing.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Identifier, "foo", 10, 13);
assert_eq!(tok.len(), 3);

pub fn is_empty(&self) -> bool

Return whether the token span is empty.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Eof, "", 8, 8);
assert!(tok.is_empty());

pub fn display_name(&self) -> &'static str

Return a human-readable display name for this token.

pub fn as_ref_token(&self) -> TokenRef<'_>

Return a borrowed token view over this token.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl From<TokenRef<'_>> for Token

fn from(value: TokenRef<'_>) -> Token

Converts to this type from the input type.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token