Crate perl_tokenizer 

Token stream and trivia utilities for the parser.

Wraps the raw token output of perl-lexer into a position-aware TokenStream and preserves whitespace/comment trivia via the TriviaPreservingParser. Used by the parser and formatting provider to maintain lossless round-trip fidelity.

Re-exports

pub use token_stream::TokenStream;

pub use token_wrapper::TokenWithPosition;

pub use trivia::Trivia;

pub use trivia::TriviaToken;

pub use trivia_parser::TriviaParserContext;

pub use trivia_parser::TriviaPreservingParser;

Modules

token_stream

Token stream adapter between perl-lexer output and the parser.

token_wrapper

Token wrapper with enhanced position tracking

trivia

Trivia (comments and whitespace) handling for the Perl parser

trivia_parser

Trivia-preserving parser implementation

util

Utility functions for the Perl parser

Structs

Token

Token produced by the lexer and consumed by the parser.

Enums

TokenKind

Token classification for Perl parsing.