Struct GraphQLTokenStream 

pub struct GraphQLTokenStream<'src, TTokenSource: GraphQLTokenSource<'src>> { /* private fields */ }

Streaming lexer that produces GraphQLTokens given some GraphQLTokenSource with a bounded lookahead buffer.

This structure accepts any GraphQLTokenSource and provides lookahead capabilities while maintaining efficient streaming behavior. It centralizes buffering, peeking, and lookahead logic.

Since trivia is already attached to tokens by the lexer, the parser can simply call peek() and consume() without worrying about trivia.

Internal Buffer Management

Tokens are stored in a VecDeque ring buffer. Unconsumed tokens are buffered at the back; consume() pops from the front and returns the owned token via O(1) pop_front().

Type Parameters

• 'src - The lifetime of the source text that tokens are lexed from.
• TTokenSource - The underlying token source, which must implement GraphQLTokenSource (i.e., Iterator<Item = GraphQLToken>).

Future TODOs

• Consider adding a GraphQLTokenStreamOptions struct to configure behavior:
  • include_trivia: bool - Whether to include preceding_trivia in tokens (can be disabled for performance when trivia is not needed)
  • max_tokens: Option<usize> - Limit total tokens returned (DoS protection)

Implementations

impl<'src, TTokenSource: GraphQLTokenSource<'src>> GraphQLTokenStream<'src, TTokenSource>

pub fn consume(&mut self) -> Option<GraphQLToken<'src>>

Advance to the next token and return it as an owned value.

Returns None if the stream is exhausted.

pub fn current_buffer_len(&self) -> usize

Returns the number of GraphQLTokens currently buffered (unconsumed).

pub fn is_at_end(&mut self) -> bool

Check if we’ve reached the end of the stream.

Returns true if there are no more tokens to consume, or if the next token is Eof.

pub fn new(token_source: TTokenSource) -> Self

Creates a new token stream from a token source.

pub fn peek(&mut self) -> Option<&GraphQLToken<'src>>

Peek at the next token without consuming it.

Returns the front of the buffer (filling it first if empty). Returns None if the stream is exhausted.

pub fn peek_nth(&mut self, n: usize) -> Option<&GraphQLToken<'src>>

Peek at the nth token ahead (0-indexed from next unconsumed token).

peek_nth(0) is equivalent to peek().

Fills the buffer up to n+1 elements if needed. Returns None if the stream ends before reaching position n.