Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn tokenize<'a>(
        &'a self,
        input_text: &'a str,
    ) -> impl Iterator<Item = String> + 'a;
}

A tokenizer splits text into a sequence of tokens. Implement this trait to use this crate with your own tokenizer.

Required Methods

fn tokenize<'a>( &'a self, input_text: &'a str, ) -> impl Iterator<Item = String> + 'a

Tokenizes the input text.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors