Struct VocabularyHandle 

pub struct VocabularyHandle { /* private fields */ }

An immutable, opaque handle owning ONE canonical vocabulary representation and its content fingerprint - always computed here, never caller-suppliable.

Implementations

impl VocabularyHandle

pub fn new(vocab: Arc<Vocabulary>) -> Result<Self, CompileError>

Wraps vocab, building the canonical representation once, admitted against the shared build-memory gate so concurrent large constructions cannot each exceed the process budget.

pub fn new_with_logits_vocab_size( vocab: Arc<Vocabulary>, logits_vocab_size: Option<usize>, ) -> Result<Self, CompileError>

new, with an explicit mask width instead of the inferred one. Rejected if Some and smaller than the largest token/EOS id actually present.

pub fn new_admitted( vocab: Arc<Vocabulary>, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

new, but the caller already holds a PermitGuard for this whole operation - re-verified against this route’s own estimate first, so an undersized permit is rejected, not trusted.

pub fn new_admitted_with_logits_vocab_size( vocab: Arc<Vocabulary>, logits_vocab_size: Option<usize>, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

new_admitted, with the same explicit-mask-width contract as new_with_logits_vocab_size.

pub fn from_packed( token_bytes: &[u8], byte_offsets: &[u32], token_ids: &[u32], id_offsets: &[u32], eos_token_id: u32, ) -> Result<Self, CompileError>

Builds directly from four packed CSR buffers (the tuple-free fast path): no intermediate Vocabulary/HashMap. Same validation as new, admitted before any construction runs.

pub fn from_packed_with_logits_vocab_size( token_bytes: &[u8], byte_offsets: &[u32], token_ids: &[u32], id_offsets: &[u32], eos_token_id: u32, logits_vocab_size: Option<usize>, ) -> Result<Self, CompileError>

from_packed, with the same explicit-mask-width contract as new_with_logits_vocab_size.

pub fn from_packed_admitted( token_bytes: &[u8], byte_offsets: &[u32], token_ids: &[u32], id_offsets: &[u32], eos_token_id: u32, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

from_packed, but the caller already holds a PermitGuard covering the whole ingestion - re-verified the same way as new_admitted, from the actual slices.

pub fn from_packed_admitted_with_logits_vocab_size( token_bytes: &[u8], byte_offsets: &[u32], token_ids: &[u32], id_offsets: &[u32], eos_token_id: u32, logits_vocab_size: Option<usize>, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

from_packed_admitted, with the same explicit-mask-width contract as new_with_logits_vocab_size.

pub fn from_dense_id_ordered( token_bytes: &[u8], byte_offsets: &[u32], present: &[bool], eos_token_id: u32, ) -> Result<Self, CompileError>

Builds directly from a DENSE, ID-ORDERED tokenizer decode: decoded[i] (via byte_offsets) is token id i’s bytes, present[i] says whether slot i exists. No id list, no

pub fn from_dense_id_ordered_with_logits_vocab_size( token_bytes: &[u8], byte_offsets: &[u32], present: &[bool], eos_token_id: u32, logits_vocab_size: Option<usize>, ) -> Result<Self, CompileError>

from_dense_id_ordered, with the same explicit-mask-width contract as new_with_logits_vocab_size.

pub fn from_dense_id_ordered_admitted( token_bytes: &[u8], byte_offsets: &[u32], present: &[bool], eos_token_id: u32, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

from_dense_id_ordered, but the caller already holds a PermitGuard covering the whole ingestion - re-verified from the actual slices, same as from_packed_admitted.

pub fn from_dense_id_ordered_admitted_with_logits_vocab_size( token_bytes: &[u8], byte_offsets: &[u32], present: &[bool], eos_token_id: u32, logits_vocab_size: Option<usize>, permit: &PermitGuard<'_>, ) -> Result<Self, CompileError>

from_dense_id_ordered_admitted, with the same explicit-mask-width contract as new_with_logits_vocab_size.

pub fn fingerprint(&self) -> VocabFingerprint

The vocabulary’s content fingerprint, computed once at construction.

pub fn mask_vocab_size(&self) -> usize

The mask width (max(largest token id, EOS id) + 1, or the explicit logits_vocab_size this handle was built with) - the number of bits one mask row covers.

pub fn eos_token_id(&self) -> u32

The EOS token id (a side value, never a mask bit).

pub fn len(&self) -> usize

The number of stored ordinary token ids plus one for EOS (NOT the distinct byte-string count and NOT the mask width).

pub fn is_empty(&self) -> bool

True when the token map has no records (matches Vocabulary::is_empty()).

pub fn token_bytes(&self, id: u32) -> Option<&[u8]>

id’s byte string, or None if id names no ordinary token (including EOS).

pub fn iter_records(&self) -> impl Iterator<Item = (&[u8], &[u32])>

Every record as (bytes, ids), byte-sorted (for a caller with no precomputed mask table).

pub fn token_slice_retained_bytes(&self) -> usize

Bytes retained by the optional vocabulary-derived token-slice catalogue.

Trait Implementations

impl Clone for VocabularyHandle

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VocabularyHandle

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.