Trait summavy_tokenizer_api::TokenStream

pub trait TokenStream {
    fn advance(&mut self) -> bool;
    fn token(&self) -> &Token;
    fn token_mut(&mut self) -> &mut Token;

    fn next(&mut self) -> Option<&Token> { ... }
    fn process(&mut self, sink: &mut dyn FnMut(&Token)) { ... }
}

TokenStream is the result of the tokenization.

It consists consumable stream of Tokens.

Required Methods

fn advance(&mut self) -> bool

Advance to the next token

Returns false if there are no other tokens.

fn token(&self) -> &Token

Returns a reference to the current token.

fn token_mut(&mut self) -> &mut Token

Returns a mutable reference to the current token.

Provided Methods

fn next(&mut self) -> Option<&Token>

Helper to iterate over tokens. It simply combines a call to .advance() and .token().

fn process(&mut self, sink: &mut dyn FnMut(&Token))

Helper function to consume the entire TokenStream and push the tokens to a sink function.

Trait Implementations

impl<'a> TokenStream for Box<dyn TokenStream + 'a>

fn advance(&mut self) -> bool

Advance to the next token

fn token<'b>(&'b self) -> &'b Token

Returns a reference to the current token.

fn token_mut<'b>(&'b mut self) -> &'b mut Token

Returns a mutable reference to the current token.

fn next(&mut self) -> Option<&Token>

Helper to iterate over tokens. It simply combines a call to .advance() and .token().

fn process(&mut self, sink: &mut dyn FnMut(&Token))

Helper function to consume the entire TokenStream and push the tokens to a sink function.

Implementations on Foreign Types

impl<'a> TokenStream for Box<dyn TokenStream + 'a>

fn advance(&mut self) -> bool

fn token<'b>(&'b self) -> &'b Token

fn token_mut<'b>(&'b mut self) -> &'b mut Token

Implementors