Skip to main content

Tokenizer

kopitiam_tokenizer

Trait Tokenizer 

Source 
pub trait Tokenizer {
    // Required methods
    fn encode(&self, text: &str) -> Result<Vec<u32>>;
    fn decode(&self, ids: &[u32]) -> Result<String>;
    fn vocab_size(&self) -> usize;
}
Expand description

What every tokenizer implementation in the Kopitiam Runtime must provide. kopitiam-runtime codes against this trait, not BpeTokenizer directly, so a future tokenizer scheme (e.g. SentencePiece for a model family that needs it) can be dropped in without touching the runtime’s call sites.

Required Methods§

Source

fn encode(&self, text: &str) -> Result<Vec<u32>>

Encodes text into token ids. Byte-level implementations never fail here – every possible &str (any valid UTF-8, including emoji, CJK, and arbitrary punctuation runs) is representable as some sequence of byte-derived tokens – so this returning Result is about leaving room for non-byte-level implementations, not because BpeTokenizer::encode can actually fail today.

Source

fn decode(&self, ids: &[u32]) -> Result<String>

Decodes token ids back into text. Fails gracefully (rather than panicking) on an id outside the vocab; never fails just because the concatenated bytes happen not to be valid UTF-8 on their own (see bpe::BpeTokenizer::decode’s docs for why that case is handled losslessly-in-practice-but-not-by-contract via lossy UTF-8 conversion instead of an error).

Source

fn vocab_size(&self) -> usize

Total number of ids in the vocabulary, including any registered special tokens.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§