Skip to main content

TextTokenizer

ferrox_models::engine

Trait TextTokenizer 

Source 
pub trait TextTokenizer {
    // Required methods
    fn encode(&self, text: &str) -> Vec<usize>;
    fn decode(&self, ids: &[usize]) -> String;

    // Provided method
    fn decode_bytes(&self, ids: &[usize]) -> Vec<u8> ⓘ { ... }
}
Expand description

A minimal text<->token-id interface shared by every real tokenizer this crate has, regardless of each one’s native id width (GgufBpeTokenizer/GgufSpmTokenizer/GgufUnigramTokenizer use u32, KimiTokenizer also uses u32) – lets a generic generation loop encode/decode without caring which concrete tokenizer it was given.

Required Methods§

Source

fn encode(&self, text: &str) -> Vec<usize>

Source

fn decode(&self, ids: &[usize]) -> String

Provided Methods§

Source

fn decode_bytes(&self, ids: &[usize]) -> Vec<u8> ⓘ

The raw bytes, before any UTF-8 decision is made about them.

A caller decoding ONE token at a time needs these: a character split across two tokens is two invalid fragments, and decode resolves each to U+FFFD separately, losing the bytes (#124).

The default is correct for any tokenizer whose tokens are whole text, and no worse than decode for one whose tokens are not – but such a tokenizer should override this.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§