Struct CompiledVocabulary 

pub struct CompiledVocabulary { /* private fields */ }

A prepared, reusable vocabulary; build once and reuse across binds. Cheap to clone.

Implementations

impl CompiledVocabulary

pub fn new(vocabulary: Arc<Vocabulary>) -> Result<Self, CompileError>

Wraps vocabulary, inferring the mask width from its largest token/EOS id.

pub fn with_logits_vocab_size( vocabulary: Arc<Vocabulary>, logits_vocab_size: usize, ) -> Result<Self, CompileError>

new, with an explicit mask width (e.g. a model whose logits are padded wider than the tokenizer’s own id range). Rejected if narrower than the largest token/EOS id present.

pub fn from_packed( token_bytes: &[u8], byte_offsets: &[u32], token_ids: &[u32], id_offsets: &[u32], eos_token_id: u32, ) -> Result<Self, CompileError>

Builds directly from four packed CSR buffers, skipping an intermediate Vocabulary/HashMap.

pub fn from_handle(handle: VocabularyHandle) -> Self

Wraps an already-built VocabularyHandle, skipping re-canonicalization.

pub fn mask_vocab_size(&self) -> usize

The mask width: write_mask buffers must hold ceil(mask_vocab_size / 32) words.

pub fn eos_token_id(&self) -> u32

This vocabulary’s EOS token id.

pub fn token_bytes(&self, token_id: u32) -> Option<&[u8]>

Exact raw bytes associated with token_id, or None for an absent/special id.

Trait Implementations

impl Clone for CompiledVocabulary

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl TryFrom<Arc<Vocabulary>> for CompiledVocabulary

type Error = CompileError

The type returned in the event of a conversion error.

fn try_from(vocabulary: Arc<Vocabulary>) -> Result<Self, Self::Error>

Performs the conversion.