Skip to main content

Tokenizer

khive_text

Trait Tokenizer 

Source 
pub trait Tokenizer: Send + Sync {
    // Required method
    fn tokenize(&self, text: &str) -> Vec<String>;
}
Expand description

Splits a string into raw tokens. Must be deterministic and stateless.

Required Methods§

Source

fn tokenize(&self, text: &str) -> Vec<String>

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§