Skip to main content

TextTokenizer

ferrox_models::engine

Trait TextTokenizer 

Source 
pub trait TextTokenizer {
    // Required methods
    fn encode(&self, text: &str) -> Vec<usize>;
    fn decode(&self, ids: &[usize]) -> String;
}
Expand description

A minimal text<->token-id interface shared by every real tokenizer this crate has, regardless of each one’s native id width (GgufBpeTokenizer/GgufSpmTokenizer/GgufUnigramTokenizer use u32, KimiTokenizer also uses u32) – lets a generic generation loop encode/decode without caring which concrete tokenizer it was given.

Required Methods§

Source

fn encode(&self, text: &str) -> Vec<usize>

Source

fn decode(&self, ids: &[usize]) -> String

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§