Trait TokenizerWrapper

pub trait TokenizerWrapper {
    // Required method
    fn tokenize(&self, text: &str) -> Option<Vec<String>>;
}

TokenizerWrapper

We wrap the tokenizer for a specific embedding model to allow for a common interface for tokenization.

Required Methods

fn tokenize(&self, text: &str) -> Option<Vec<String>>

Implementors