Skip to main content

TokenizerClone

hermes_core::tokenizer

Trait TokenizerClone 

Source 
pub trait TokenizerClone: Send + Sync {
    // Required methods
    fn tokenize(&self, text: &str) -> Vec<Token>;
    fn clone_box(&self) -> BoxedTokenizer;
}

Required Methods§

Source

fn tokenize(&self, text: &str) -> Vec<Token>

Source

fn clone_box(&self) -> BoxedTokenizer

Implementors§