Struct tokenizer_lib::GeneratorTokenQueue
source · pub struct GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where
T: TokenTrait,
for<'a> TGenerator: FnMut(&mut TGeneratorState, &mut GeneratorTokenQueueBuffer<'a, T, TData>),{ /* private fields */ }
Expand description
A token queue which has a backing generator/lexer which is called when needed by parsing logic
Implementations§
source§impl<T, TData, TGeneratorState, TGenerator> GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where
T: TokenTrait,
for<'a> TGenerator: FnMut(&mut TGeneratorState, &mut GeneratorTokenQueueBuffer<'a, T, TData>),
impl<T, TData, TGeneratorState, TGenerator> GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: TokenTrait, for<'a> TGenerator: FnMut(&mut TGeneratorState, &mut GeneratorTokenQueueBuffer<'a, T, TData>),
sourcepub fn new(generator: TGenerator, generator_state: TGeneratorState) -> Self
pub fn new(generator: TGenerator, generator_state: TGeneratorState) -> Self
Create a new GeneratorTokenQueue
with a lexer function and initial state
Trait Implementations§
source§impl<T, TData, TGeneratorState, TGenerator> TokenReader<T, TData> for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where
T: TokenTrait,
TData: Debug,
for<'a> TGenerator: FnMut(&mut TGeneratorState, &mut GeneratorTokenQueueBuffer<'a, T, TData>),
impl<T, TData, TGeneratorState, TGenerator> TokenReader<T, TData> for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: TokenTrait, TData: Debug, for<'a> TGenerator: FnMut(&mut TGeneratorState, &mut GeneratorTokenQueueBuffer<'a, T, TData>),
source§fn peek(&mut self) -> Option<&Token<T, TData>>
fn peek(&mut self) -> Option<&Token<T, TData>>
Returns a reference to next token but does not advance current position
source§fn peek_n(&mut self, n: usize) -> Option<&Token<T, TData>>
fn peek_n(&mut self, n: usize) -> Option<&Token<T, TData>>
Returns a reference to nth (zero based) upcoming token without advancing
source§fn scan(
&mut self,
cb: impl FnMut(&T, &TData) -> bool
) -> Option<&Token<T, TData>>
fn scan( &mut self, cb: impl FnMut(&T, &TData) -> bool ) -> Option<&Token<T, TData>>
Runs the closure (cb) over upcoming tokens. Passes the value behind the Token to the closure.
Will stop and return a reference to the next Token from when the closure returns true.
Returns None if scanning finishes before closure returns true. Does not advance the reader. Read more
source§fn conditional_next(
&mut self,
cb: impl FnOnce(&T) -> bool
) -> Option<Token<T, TData>>
fn conditional_next( &mut self, cb: impl FnOnce(&T) -> bool ) -> Option<Token<T, TData>>
Returns next if
cb
returns true for the upcoming token (the token from TokenReader::peek)source§fn expect_next(
&mut self,
expected_type: T
) -> Result<TData, Option<(T, Token<T, TData>)>>
fn expect_next( &mut self, expected_type: T ) -> Result<TData, Option<(T, Token<T, TData>)>>
Tests that next token matches an expected type. Will return error if does not
match. The
Ok
value contains the data of the valid token.
Else it will return the Err with the expected token type and the token that did not match Read moreAuto Trait Implementations§
impl<T, TData, TGeneratorState, TGenerator> RefUnwindSafe for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: RefUnwindSafe, TData: RefUnwindSafe, TGenerator: RefUnwindSafe, TGeneratorState: RefUnwindSafe,
impl<T, TData, TGeneratorState, TGenerator> Send for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: Send, TData: Send, TGenerator: Send, TGeneratorState: Send,
impl<T, TData, TGeneratorState, TGenerator> Sync for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: Sync, TData: Sync, TGenerator: Sync, TGeneratorState: Sync,
impl<T, TData, TGeneratorState, TGenerator> Unpin for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: Unpin, TData: Unpin, TGenerator: Unpin, TGeneratorState: Unpin,
impl<T, TData, TGeneratorState, TGenerator> UnwindSafe for GeneratorTokenQueue<T, TData, TGeneratorState, TGenerator>where T: UnwindSafe, TData: UnwindSafe, TGenerator: UnwindSafe, TGeneratorState: UnwindSafe,
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more