Skip to main content

TokenizerWorker

dynamo_bench::coding::tokenizer

Trait TokenizerWorker 

Source 
pub trait TokenizerWorker: Send + 'static {
    // Required method
    fn encode(&mut self, text: &str) -> Result<Vec<u32>>;

    // Provided method
    fn encode_with_word_overlap(
        &mut self,
        text: &str,
        previous_text: Option<&str>,
        previous_tokens: Option<&[u32]>,
        overlap_words: usize,
    ) -> Result<Vec<u32>> { ... }
}

Required Methods§

Source

fn encode(&mut self, text: &str) -> Result<Vec<u32>>

Provided Methods§

Source

fn encode_with_word_overlap( &mut self, text: &str, previous_text: Option<&str>, previous_tokens: Option<&[u32]>, overlap_words: usize, ) -> Result<Vec<u32>>

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§