Struct TokenStream 

pub struct TokenStream<'a> { /* private fields */ }

Representation of the input as a sequence of tokens

The TokenStrem holds the reference of the input stream and is responsible for tokenizing it into a sequence of Tokens. TokenStream is immutable, all operations that extract tokens from the stream must return another instance of a TokenStream representing tokens remaining in the stream.

Opening and closing parenthesis have special meaning for the token streams. They are intended to enclose tokens related to the same value to avoid ambiguity when parsing nested structures with unknown number of required arguments, such as nested vectors. TokenStream aims to make handling parenthesis as transparent as possible:

• attempt to take a token from a stream fails with an UnbalancedParenthesis error if the non-consumed portion of the input starts with an opening parenthesis;
• the token stream is considered to be empty if the non-consumed portion of the input start with a closing parenthesis;
• to handle nested structures, use either with_nested or complete_nested. These methods take a closure that returns a result and a TokenStream representing the remainder of the stream that wasn’t consumed during parsing. Token stream checks that all tokens consumed by the inner closure have been processed and return an appropriate error otherwise.

Implementations

impl<'a> TokenStream<'a>

pub fn new(input: &'a str) -> Self

Creates a new TokenStream from an input string slice.

pub fn is_empty(&self) -> bool

Returns true if the last token of the stream has been consumed.

More specifically, it returns true if the non-consumed portion of the input string is empty or contains an arbitrary number of whitespace characters followed by an octothorp (#), a closing parenthesis, or the end of a string.

pub fn is_all_consumed(&self) -> bool

Returns true if the entirety of the input string was consumed and the last token is not followed by any whitespace characters or a comment.

This is useful when performing completion: suggestions should only be generated for the last token of the stream if it isn’t followed by any other character.

If is_all_consumed returns true, then is_empty must return true, while reverse is not true.

pub fn peek(&self) -> Option<Result<Token<'a>, UnbalancedParenthesis>>

Returns the next token in the token stream, if any. This is an efficient operation that does not cause any lexing operations and should be preferred when the token will not be consumed.

It returns:

• Some(Ok(token)) if the stream is not empty;
• Some(Err(UnbalancedParenthesis)) if the remaining part of the string starts with a closing parenthesis;
• None if the stream is empty.

pub fn take( &self, ) -> Option<Result<(Token<'a>, TokenStream<'a>), UnbalancedParenthesis>>

Returns the next token in the stream and an instance of the TokenStream representing the remaining tokens. The behavior is similar to peek with the same returned value with addition of a TokenStream instance.

pub fn with_nested<R, F: FnOnce(TokenStream<'a>) -> ParseResult<'a, R>>( &self, callback: F, ) -> ParseResult<'a, R>

Executes a closure on an inner portion of the token stream. The closure passed to this function returns either a parsed value along with the remaining TokenSream, or a parsing failure.

This method behaves differently depending on whether the non-consumed portion of the input string starts with an opening parenthesis. If not, the value returned by the closure as is.

Otherwise, this functions consumes the parenthesis, and calls the closure. with_nested then returns a parse failure if the closure call failed or if there are any not consumed tokens remaining. Note, that in case the closure fails due to an [UnrecognizedToken] it’s converted into an error.

pub fn complete_nested<F: FnOnce(TokenStream<'a>) -> CompletionResult<'a>>( &self, callback: F, ) -> CompletionResult<'a>

Executes a closure on an inner portion of the token stream. If the non-consumed portion of the input stream starts with an opening parenthesis, this function attempts to skip all tokens until the corresponding closing parenthesis. If no such parenthesis is found or if there is no opening parenthesis, the closure is called and its execution result is returned.

Trait Implementations

impl<'a> Clone for TokenStream<'a>

fn clone(&self) -> TokenStream<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for TokenStream<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Copy for TokenStream<'a>