Struct tokenizer_lib::BufferedTokenQueue

pub struct BufferedTokenQueue<T: TokenTrait, TData> { /* private fields */ }

A queue which can be used as a sender and reader. Use this for buffering all the tokens before reading

Implementations

impl<T: TokenTrait, TData> BufferedTokenQueue<T, TData>

pub fn new() -> Self

Constructs a new BufferedTokenQueue

Trait Implementations

impl<T: TokenTrait, TData> Default for BufferedTokenQueue<T, TData>

fn default() -> Self

Returns the “default value” for a type.

impl<T: TokenTrait, TData> TokenReader<T, TData> for BufferedTokenQueue<T, TData>

fn peek(&mut self) -> Option<&Token<T, TData>>

Returns a reference to next token but does not advance current position

fn peek_n(&mut self, n: usize) -> Option<&Token<T, TData>>

Returns a reference to nth (zero based) upcoming token without advancing

fn peek_mut(&mut self) -> Option<&mut Token<T, TData>>

Use with caution

fn next(&mut self) -> Option<Token<T, TData>>

Returns the next token and advances

fn scan( &mut self, cb: impl FnMut(&T, &TData) -> bool ) -> Option<&Token<T, TData>>

Runs the closure (cb) over upcoming tokens. Passes the value behind the Token to the closure. Will stop and return a reference to the next Token from when the closure returns true. Returns None if scanning finishes before closure returns true. Does not advance the reader.

fn conditional_next( &mut self, cb: impl FnOnce(&T) -> bool ) -> Option<Token<T, TData>>

Returns next if cb returns true for the upcoming token (the token from TokenReader::peek)

fn expect_next( &mut self, expected_type: T ) -> Result<TData, Option<(T, Token<T, TData>)>>

Tests that next token matches an expected type. Will return error if does not match. The Ok value contains the data of the valid token. Else it will return the Err with the expected token type and the token that did not match

impl<T: TokenTrait, TData> TokenSender<T, TData> for BufferedTokenQueue<T, TData>

fn push(&mut self, token: Token<T, TData>) -> bool

Appends a new Token Will return false if could not push token