Struct TokenByteTable 

pub struct TokenByteTable { /* private fields */ }

Maps every token id in a vocabulary to the raw bytes it contributes to the decoded output. Special/control tokens map to empty (they contribute no output bytes).

Implementations

impl TokenByteTable

pub fn from_tokenizer_json(tokenizer_json: &[u8]) -> Result<Self>

Build from a tokenizer.json, choosing the byte-level path when possible and the SPM fallback otherwise; refuses loudly if neither applies.

pub fn from_vocab_blob(blob: &[u8]) -> Result<Self>

Parse the engine’s id→bytes vocab blob (u32 count, then per id u32 len + bytes) into a byte table. This is the byte-level BPE path.

pub fn bytes(&self, id: u32) -> Option<&[u8]>

The raw bytes for a token id, or None if the id is out of range.

pub fn len(&self) -> usize

The number of token ids the table covers.

pub fn is_empty(&self) -> bool

Trait Implementations

impl Clone for TokenByteTable

fn clone(&self) -> TokenByteTable

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenByteTable

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.