Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub text: Arc<str>,
    pub start: usize,
    pub end: usize,
}

Token produced by the lexer and consumed by the parser.

Stores the token kind, original source text, and byte span. The text is kept in an Arc<str> so buffering and lookahead can clone tokens cheaply.

Fields

kind: TokenKind

Token classification for parser decision making

text: Arc<str>

Original source text for precise reconstruction

start: usize

Starting byte position for error reporting and location tracking

end: usize

Ending byte position for span calculation and navigation

Implementations

impl Token

pub fn new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Token

Create a new token with the given kind, source text, and byte span.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Sub, "sub", 0, 3);
assert_eq!(tok.kind, TokenKind::Sub);
assert_eq!(&*tok.text, "sub");

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token