perl-tokenizer

Token stream, trivia preservation, and utility functions for the Perl parser pipeline.

Overview

perl-tokenizer bridges the raw perl-lexer output and the recursive descent parser. It provides a buffered token stream with up to three-token lookahead, trivia-preserving lexing and parsing for code formatting, position tracking wrappers, and helpers for locating __DATA__/__END__ markers.

Public API

• TokenStream -- parser-oriented stream with peek, peek_second, peek_third, and statement-boundary reset
• TokenWithPosition / PositionTracker -- attach line/column info to lexer tokens
• Trivia, TriviaToken, TriviaLexer -- classify and collect comments, whitespace, and POD as trivia
• TriviaPreservingParser, TriviaParserContext -- parse source while preserving trivia on AST nodes
• find_data_marker_byte_lexed, code_slice -- locate __DATA__/__END__ boundaries
• Re-exports Token and TokenKind from perl-token

Workspace Role

Internal Tier 2 crate consumed by perl-parser-core and downstream tooling. Not intended for standalone use outside the workspace.

License

MIT OR Apache-2.0