Struct RawTokenizer 

pub struct RawTokenizer { /* private fields */ }

Trait Implementations

impl Clone for RawTokenizer

fn clone(&self) -> RawTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for RawTokenizer

fn default() -> RawTokenizer

Returns the “default value” for a type.

impl Tokenizer for RawTokenizer

type TokenStream<'a> = RawTokenStream<'a>

Stream type emitted by the tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.