Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn tokenize(&self, input_text: &str) -> Vec<String>;
}

A tokenizer splits text into a sequence of tokens. Implement this trait to use this crate with your own tokenizer.

Required Methods

fn tokenize(&self, input_text: &str) -> Vec<String>

Tokenizes the input text.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors