Struct TokenStream 

pub struct TokenStream<'a> { /* private fields */ }

Token types and token stream for lexer output. Token stream that wraps perl-lexer

Implementations

impl<'a> TokenStream<'a>

pub fn new(input: &'a str) -> TokenStream<'a>

Create a new token stream from source code

pub fn peek(&mut self) -> Result<&Token, ParseError>

Peek at the next token without consuming it

pub fn next(&mut self) -> Result<Token, ParseError>

Consume and return the next token

pub fn is_eof(&mut self) -> bool

Check if we’re at the end of input

pub fn peek_second(&mut self) -> Result<&Token, ParseError>

Peek at the second token (two tokens ahead)

pub fn peek_third(&mut self) -> Result<&Token, ParseError>

Peek at the third token (three tokens ahead)

pub fn enter_format_mode(&mut self)

Enter format body parsing mode in the lexer

pub fn on_stmt_boundary(&mut self)

Called at statement boundaries to reset lexer state and clear cached lookahead

pub fn relex_as_term(&mut self)

Re-lex the current peeked token in ExpectTerm mode.

This is needed for context-sensitive constructs like split /regex/ where the / was lexed as division (Slash) but should be a regex delimiter. Rolls the lexer back to the peeked token’s start position, switches to ExpectTerm mode, and clears the peek cache so the next peek() or next() re-lexes it as a regex.

pub fn invalidate_peek(&mut self)

Pure peek cache invalidation - no mode changes

pub fn peek_fresh_kind(&mut self) -> Option<TokenKind>

Convenience method for a one-shot fresh peek