Struct PtxTokenStream 

pub struct PtxTokenStream<'a> { /* private fields */ }

Token stream wrapper for parsing PTX tokens.

This struct provides methods for consuming and inspecting tokens during parsing.

Implementations

impl<'a> PtxTokenStream<'a>

pub fn new(tokens: &'a [(PtxToken, Span)]) -> Self

pub fn peek(&self) -> Result<&'a (PtxToken, Span), PtxParseError>

Peek at the next token without consuming it.

Behavior for complete mode

Returns the token at the current stream position without advancing the position. This is a simple array lookup at index.0.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This method only operates on whole tokens and cannot be used during partial (character-by-character) matching mode.

Returns

• Ok(&(PtxToken, Span)) - The token and its span
• Err(PtxParseError) - If at end of stream (UnexpectedEof) or in partial mode (InvalidModeForTokenMethod)

pub fn peek_n( &self, offset: usize, ) -> Result<&'a (PtxToken, Span), PtxParseError>

Peek at the token offset positions ahead without consuming it.

Behaves like peek() but allows inspecting future tokens in complete mode.

pub fn consume(&mut self) -> Result<&'a (PtxToken, Span), PtxParseError>

Consume and return the next token.

Behavior for complete mode

Advances the stream position by one token (increments index.0). Returns the token that was at the current position before advancing.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This method only operates on whole tokens and cannot be used during partial (character-by-character) matching mode.

Returns

• Ok(&(PtxToken, Span)) - The consumed token and its span
• Err(PtxParseError) - If at end of stream (UnexpectedEof) or in partial mode (InvalidModeForTokenMethod)

pub fn consume_if<F>(&mut self, predicate: F) -> Option<&'a (PtxToken, Span)>

where F: FnOnce(&PtxToken) -> bool,

Conditionally consume the next token if it matches the predicate.

Returns

• Some(&(PtxToken, Span)) - If the predicate returns true, consumes and returns the token
• None - If the predicate returns false or if at end of stream

pub fn expect( &mut self, expected: &PtxToken, ) -> Result<&'a (PtxToken, Span), PtxParseError>

Check if the next token is the expected type, and if so, consume it. Otherwise, return an error and do NOT consume the token.

Behavior for complete mode

Peeks at the current token and checks if its discriminant (variant type) matches the expected token discriminant. If it matches, advances the stream by one token and returns the token. If it doesn’t match, returns an UnexpectedToken error without consuming anything.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This method only operates on whole tokens and cannot be used during partial (character-by-character) matching mode.

Returns

• Ok(&(PtxToken, Span)) - The matched and consumed token
• Err(PtxParseError) - If token doesn’t match (UnexpectedToken) or in partial mode (InvalidModeForTokenMethod)

pub fn expect_identifier(&mut self) -> Result<(String, Span), PtxParseError>

Check if the next token is an identifier, and if so, consume it and return the String.

Behavior for complete mode

Expects the current token to be an Identifier, consumes it, and returns its string value.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This is a token-based method.

pub fn expect_register(&mut self) -> Result<(String, Span), PtxParseError>

Check if the next token is a register, and if so, consume it and return the String.

Behavior for complete mode

Expects the current token to be a Register, consumes it, and returns its string value.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This is a token-based method.

pub fn expect_directive(&mut self) -> Result<(String, Span), PtxParseError>

Check if the next token is a directive (Dot + Identifier), and if so, consume them and return the String.

Behavior for complete mode

Expects a Dot token followed by an Identifier token, consumes both, and returns the identifier string with a combined span covering both tokens.

Behavior for partial mode

Returns an error (InvalidModeForTokenMethod). This is a token-based method.

pub fn expect_strings( &mut self, candidates: &[&str], ) -> Result<usize, PtxParseError>

Try to match and consume a sequence of tokens that matches one of the candidate strings. Returns the index of the matched candidate.

This is used for parsing modifiers that may contain :: sequences like “.to::cluster” The candidates should include the leading dot (e.g., [“.to::cluster”, “.to::cta”])

Behavior for complete mode

Tries to match each candidate string against the token stream by consuming whole tokens. Returns the index of the first candidate that matches. Uses backtracking (position/set_position) to try each candidate without consuming tokens on failed attempts.

Behavior for partial mode

Supports character-by-character matching within tokens using the char offset. This allows matching patterns that span across token boundaries or within tokens. Uses backtracking to restore position when a candidate fails to match.

pub fn expect_string(&mut self, expected: &str) -> Result<(), PtxParseError>

Expect that the next sequence of tokens matches the given string pattern.

Behavior for complete mode

Matches the pattern against the token stream by consuming whole tokens. Each token’s string representation must match consecutive characters in the pattern. The match succeeds only if the entire pattern is consumed and tokens are fully consumed.

Behavior for partial mode

Matches the pattern character-by-character against the token stream using the character offset for partial token matching. This allows matching patterns that don’t align with token boundaries. If all characters match, the stream advances. If any character fails to match, the stream position is restored.

Returns

• Ok(()) if the entire pattern was successfully matched (consumed)
• Err(PtxParseError) if matching failed (UnexpectedToken)

pub fn expect_complete(&mut self) -> Result<(), PtxParseError>

Ensure we’re in complete mode (not in partial token mode). This is a no-op in complete mode, and succeeds as long as we’re not mid-token. Used by generated parsers to enforce token boundaries.

pub fn with_partial_token_mode<F, R>( &mut self, f: F, ) -> Result<R, PtxParseError>

where F: FnOnce(&mut PtxTokenStream<'_>) -> Result<R, PtxParseError>,

Execute a function in partial token mode, enabling character-by-character matching.

Behavior

This method switches the stream from complete mode to partial mode by setting the character offset to Some(0). While in partial mode, string-based methods like expect_string() can match patterns character-by-character within tokens.

After the closure completes:

• If the char offset is non-zero, validates that the current token was fully consumed
• If not fully consumed, reverts to the starting position and returns an error
• Always resets the mode back to complete mode (sets index.1 to None)

Errors

Returns an error if:

• The closure returns an error
• The token was partially consumed but not completely consumed (incomplete match)

Panics

Panics if already in partial mode (char offset is already Some).

pub fn try_with_span<F, R>(&mut self, f: F) -> Result<(R, Span), PtxParseError>

where F: FnOnce(&mut PtxTokenStream<'_>) -> Result<R, PtxParseError>,

Execute a closure with automatic backtracking and span tracking.

Saves the current stream position before running f. If f returns an error, the stream position (including partial-mode offsets) is restored. When f succeeds, this returns the closure result together with the span covering the consumed source range.

pub fn position(&self) -> StreamPosition

Get the current position in the stream, for backtracking.

Behavior for complete mode

Returns a StreamPosition containing the token index (index.0). The char offset (index.1) will be None.

Behavior for partial mode

Returns a StreamPosition containing both the token index (index.0) and the character offset within that token (index.1 = Some(offset)).

This position can be used with set_position() to restore the exact state, including the parsing mode and character offset.

pub fn set_position(&mut self, pos: StreamPosition)

Reset the stream to a previously saved position, for backtracking.

Behavior for complete mode

Restores the token index to the saved position. If the saved position was in complete mode (char offset = None), stays in complete mode.

Behavior for partial mode

Can restore to either complete or partial mode depending on the saved position. If the saved position was in partial mode (char offset = Some(n)), switches to partial mode at that exact character offset. This allows proper backtracking during character-by-character matching attempts.

pub fn is_at_end(&self) -> bool

Check if we’ve reached the end of the token stream.

Behavior for complete mode

Returns true if the token index is at or past the end of the tokens array and we’re in complete mode (char offset is None).

Behavior for partial mode

Always returns false while in partial mode (char offset is Some), even if positioned at the last token. This is because partial mode implies we’re still potentially consuming characters from the current token.

pub fn current_span(&self) -> Span

Create a zero-length span at the current stream position.