Struct Token 

pub struct Token {
    pub kind: TokenKind,
    pub text: Arc<str>,
    pub start: usize,
    pub end: usize,
}

Token types and token stream for lexer output. Token produced by the lexer and consumed by the parser.

Stores the token kind, original source text, and byte span. The text is kept in an Arc<str> so buffering and lookahead can clone tokens cheaply.

Fields

kind: TokenKind

Token classification for parser decision making

text: Arc<str>

Original source text for precise reconstruction

start: usize

Starting byte position for error reporting and location tracking

end: usize

Ending byte position for span calculation and navigation

Implementations

impl Token

pub fn new( kind: TokenKind, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Token

Create a new token with the given kind, source text, and byte span.

Examples

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::Sub, "sub", 0, 3);
assert_eq!(tok.kind, TokenKind::Sub);
assert_eq!(&*tok.text, "sub");

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token