Skip to main content

Tokenizer

hermes_core::tokenizer

Trait Tokenizer 

Source 
pub trait Tokenizer:
    Send
    + Sync
    + Clone
    + 'static {
    // Required method
    fn tokenize(&self, text: &str) -> Vec<Token>;
}
Expand description

Trait for tokenizers

Required Methods§

Source

fn tokenize(&self, text: &str) -> Vec<Token>

Tokenize the input text into a vector of tokens

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§