Crate perl_token 

Perl token definitions shared across the parser ecosystem.

This crate defines Token and TokenKind, the fundamental types that flow from the lexer (perl-lexer) into the parser (perl-parser-core). Downstream crates re-export these types so consumers rarely need to depend on perl-token directly.

Examples

Create and inspect tokens:

use perl_token::{Token, TokenKind};

// Create a keyword token for `my`
let token = Token::new(TokenKind::My, "my", 0, 2);
assert_eq!(token.kind, TokenKind::My);
assert_eq!(&*token.text, "my");
assert_eq!(token.start, 0);
assert_eq!(token.end, 2);

// Create a numeric literal token
let num = Token::new(TokenKind::Number, "42", 7, 9);
assert_eq!(num.kind, TokenKind::Number);
assert_eq!(&*num.text, "42");

Use TokenKind::display_name for user-facing error messages:

use perl_token::TokenKind;

assert_eq!(TokenKind::LeftBrace.display_name(), "'{'");
assert_eq!(TokenKind::Identifier.display_name(), "identifier");
assert_eq!(TokenKind::Eof.display_name(), "end of input");

Structs

Token

Token produced by the lexer and consumed by the parser.

TokenKindMetadata

Metadata associated with each TokenKind variant.

TokenRef

Borrowed view over token data for allocation-sensitive paths.

TokenSpan

Byte span carried by a Token.

Enums

TokenCategory

Broad classification used for token metadata and conformance checks.

TokenKind

Token classification for Perl parsing.

TokenSpanError

Error type for checked token/span constructors.