perl-token

Core token type definitions for the Perl parser ecosystem.

Overview

perl-token is a Tier 1 leaf crate that defines the shared token types used across the lexer, tokenizer, and parser crates. It has zero external dependencies (only std::sync::Arc).

Stability Contract

perl-token is a tiny stable leaf crate and should stay dependency-free at runtime. Public Token/TokenKind source compatibility is intentionally conservative and should only change with explicit, reviewed intent.

TokenKind variants: 132
Conformance update rule: adding a TokenKind variant must also update metadata, this crate's docs, and the conformance guard tests.

Public API

Token -- a token with kind: TokenKind, text: Arc<str>, start: usize, end: usize
TokenRef<'src> -- borrowed token view with kind, text: &'src str, start, end for allocation-sensitive paths
TokenKind -- enum classifying every Perl token: keywords, operators, delimiters, literals, sigils, and special tokens

Usage

use perl_token::{Token, TokenKind, TokenRef};

let tok = Token::new(TokenKind::Identifier, "foo", 0, 3);
assert_eq!(tok.kind, TokenKind::Identifier);

let borrowed = TokenRef::new(TokenKind::Identifier, "foo", 0, 3);
let owned = borrowed.to_owned_token();
assert_eq!(owned, tok);

Borrowed view from owned tokens:

use perl_token::{Token, TokenKind};

let tok = Token::new(TokenKind::My, "my", 0, 2);
let borrowed = tok.as_ref_token();
assert_eq!(borrowed.text, "my");

Benchmark scorecard

Run the token allocation scorecard benchmark:

cargo bench -p perl-token --bench token_scorecard

The benchmark group includes:

token/borrowed_construction (no Arc allocation)
token/owned_construction (Arc<str> allocation path)
token/borrowed_to_owned_conversion (explicit conversion cost)

Workspace Role

Foundational crate consumed by perl-lexer, perl-tokenizer, perl-parser-core, and downstream parser/LSP crates. Part of the tree-sitter-perl-rs workspace.

License

MIT OR Apache-2.0

perl-token 0.13.3