Struct TokenStream 

pub struct TokenStream { /* private fields */ }

An immutable, cloneable view of a token stream with a cursor position

Implementations

impl TokenStream

pub fn new(tokens: Vec<Token>) -> Self

Create a new token stream from a vector of tokens

pub fn peek(&self) -> Option<&Token>

Peek at the current token without advancing

pub fn next(self) -> (Option<Token>, TokenStream)

Consume the current token, returning it and the advanced stream

pub fn is_empty(&self) -> bool

Return true if no tokens remain (excluding Eof sentinel)

pub fn remaining(&self) -> usize

Number of tokens remaining (excluding any trailing Eof token)

pub fn position(&self) -> usize

Current cursor position (index into token slice)

pub fn byte_offset(&self) -> usize

Return the byte offset of the current token (or end-of-source)

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.