Struct SimpleTokenizer 

pub struct SimpleTokenizer { /* private fields */ }

Trait Implementations

impl Clone for SimpleTokenizer

fn clone(&self) -> SimpleTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for SimpleTokenizer

fn default() -> SimpleTokenizer

Returns the “default value” for a type.

impl Tokenizer for SimpleTokenizer

type TokenStream<'a> = SimpleTokenStream<'a>

Stream type emitted by the tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.