Module token_stream 

Token stream and trivia utilities for the parser. Buffered token stream over the raw lexer (with trivia skipping). Token stream adapter between perl-lexer output and the parser.

Provides buffered lookahead, skips trivia tokens, and resets lexer mode at statement boundaries. This stream is optimized for parser consumption rather than full-fidelity token preservation.

Basic usage

use perl_parser_core::tokens::token_stream::{TokenKind, TokenStream};

let mut stream = TokenStream::new("my $x = 42;");
assert!(matches!(stream.peek(), Ok(token) if token.kind == TokenKind::My));

while let Ok(token) = stream.next() {
    if token.kind == TokenKind::Eof {
        break;
    }
}

Pre-lexed token stream

For incremental parsing, use [TokenStream::from_vec] to create a stream from pre-lexed tokens without re-lexing from source:

use perl_parser_core::tokens::token_stream::{Token, TokenKind, TokenStream};

let tokens = vec![
    Token::new(TokenKind::My, "my", 0, 2),
    Token::new(TokenKind::ScalarSigil, "$", 3, 4),
    Token::new(TokenKind::Identifier, "x", 4, 5),
    Token::new(TokenKind::Assign, "=", 6, 7),
    Token::new(TokenKind::Number, "1", 8, 9),
    Token::new(TokenKind::Semicolon, ";", 9, 10),
    Token::new(TokenKind::Eof, "", 10, 10),
];
let mut stream = TokenStream::from_vec(tokens);
assert!(matches!(stream.peek(), Ok(t) if t.kind == TokenKind::My));

Structs

Token

Token produced by the lexer and consumed by the parser.

TokenStream

Token stream that wraps perl-lexer or a pre-lexed token buffer.

Enums

TokenKind

Token classification for Perl parsing.