Struct WhitespaceTokenizer 

pub struct WhitespaceTokenizer { /* private fields */ }

Trait Implementations

impl Clone for WhitespaceTokenizer

fn clone(&self) -> WhitespaceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for WhitespaceTokenizer

fn default() -> WhitespaceTokenizer

Returns the “default value” for a type.

impl Tokenizer for WhitespaceTokenizer

type TokenStream<'a> = WhitespaceTokenStream<'a>

Stream type emitted by the tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.