Skip to main content

TokenVocab

kopitiam_runtime

Trait TokenVocab 

Source 
pub trait TokenVocab {
    // Required methods
    fn token_count(&self) -> usize;
    fn token_bytes(&self, id: u32) -> Option<&[u8]>;
}
Expand description

A token id -> its byte string, for constraints that must reason about the bytes a candidate token would append (not just its id).

JsonStructure needs this: whether emitting token 512 keeps the JSON valid depends entirely on what characters token 512 is (":" vs , vs abc), which the id alone does not tell you. A real integration hands a view over the model’s tokenizer vocabulary here; tests use SliceVocab.

Required Methods§

Source

fn token_count(&self) -> usize

How many token ids exist (0..token_count() is the id range).

Source

fn token_bytes(&self, id: u32) -> Option<&[u8]>

The bytes token id decodes to, or None for an out-of-range / byte-less control id (e.g. EOS). A None token contributes no bytes to the grammar and is never structurally allowed by JsonStructure on its own — permit it explicitly via JsonStructure::with_always_allowed if it should be samplable (EOS usually should, so the model can actually stop).

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§