Trait TokenStream 

pub trait TokenStream: Sized {
    type Token: Clone;
    type Span: SpanLike;
    type Spanned<T: Clone>: SpannedLike<T, Span = Self::Span>;

    // Required methods
    fn peek_token_raw(&self) -> Option<&Self::Spanned<Self::Token>>;
    fn next_raw(&mut self) -> Option<Self::Spanned<Self::Token>>;
    fn cursor(&self) -> usize;
    fn rewind(&mut self, pos: usize);
    fn fork(&self) -> Self;
    fn cursor_span(&self) -> Option<Self::Span>;
    fn last_span(&self) -> Option<Self::Span>;
    fn span_at(&self, pos: usize) -> Option<Self::Span>;

    // Provided methods
    fn peek_token(&self) -> Option<&Self::Spanned<Self::Token>> { ... }
    fn next(&mut self) -> Option<Self::Spanned<Self::Token>> { ... }
    fn peek<T: Peek<Token = Self::Token>>(&self) -> bool { ... }
    fn parse<T: Parse<Token = Self::Token>>(&mut self) -> Result<T, T::Error> { ... }
    fn parse_spanned<T: Parse<Token = Self::Token> + Clone>(
        &mut self,
    ) -> Result<Self::Spanned<T>, T::Error> { ... }
    fn is_empty(&self) -> bool { ... }
    fn remaining(&self) -> usize { ... }
    fn ensure_consumed(&self) -> Result<(), Error> { ... }
    fn span_range(&self, range: Range<usize>) -> Self::Span { ... }
}

A stream of tokens for parsing.

Provides the core interface for lexer output consumption. Token streams support peeking, consumption, forking for lookahead, and rewinding.

Required Associated Types

type Token: Clone

The token type produced by the lexer.

type Span: SpanLike

The span type for tracking source locations.

type Spanned<T: Clone>: SpannedLike<T, Span = Self::Span>

A spanned wrapper type for associating values with spans.

Required Methods

fn peek_token_raw(&self) -> Option<&Self::Spanned<Self::Token>>

Peeks at the next token without consuming (includes whitespace).

fn next_raw(&mut self) -> Option<Self::Spanned<Self::Token>>

Consumes and returns the next token (includes whitespace).

fn cursor(&self) -> usize

Returns the current cursor position.

fn rewind(&mut self, pos: usize)

Rewinds to a previous cursor position.

fn fork(&self) -> Self

Creates a fork for lookahead without consuming tokens.

fn cursor_span(&self) -> Option<Self::Span>

Returns the span at the current cursor position.

fn last_span(&self) -> Option<Self::Span>

Returns the span of the last consumed token.

fn span_at(&self, pos: usize) -> Option<Self::Span>

Get the span of a token at a specific cursor position.

Returns None if the position is out of bounds.

Provided Methods

fn peek_token(&self) -> Option<&Self::Spanned<Self::Token>>

Peeks at the next significant token (skips whitespace by default).

fn next(&mut self) -> Option<Self::Spanned<Self::Token>>

Consumes and returns the next significant token.

fn peek<T: Peek<Token = Self::Token>>(&self) -> bool

Checks if the next token matches type T without consuming.

fn parse<T: Parse<Token = Self::Token>>(&mut self) -> Result<T, T::Error>

Parses a value of type T from the stream.

fn parse_spanned<T: Parse<Token = Self::Token> + Clone>( &mut self, ) -> Result<Self::Spanned<T>, T::Error>

Parses a value and wraps it with its source span.

fn is_empty(&self) -> bool

fn remaining(&self) -> usize

Returns the number of remaining significant tokens (excluding whitespace).

The default implementation counts tokens via peek_token() and next(), which may be inefficient. Implementations may override this for better performance.

fn ensure_consumed(&self) -> Result<(), Error>

Ensures the stream has been fully consumed.

Returns Ok(()) if no significant tokens remain (whitespace is ignored). Returns Err(Error::StreamNotConsumed) if tokens remain.

Example

let doc: Document = stream.parse()?;
stream.ensure_consumed()?; // Error if trailing garbage

fn span_range(&self, range: Range<usize>) -> Self::Span

Create a span covering a range of cursor positions.

This is useful for tracking the span of a parsed AST node that spans multiple tokens.

Example

let start = stream.cursor();
// ... parse multiple tokens ...
let end = stream.cursor();
let span = stream.span_range(start..end);

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors