Struct TokenizerInfo 

pub struct TokenizerInfo { /* private fields */ }

TokenizerInfo contains the vocabulary, its type, and metadata used by the grammar-guided generation.

Notes:

• Tokens may be encoded differently depending on VocabType (e.g. ByteFallback uses “<0x1B>”, ByteLevel uses unicode mappings). This wrapper exposes the decoded vocabulary in the same form as the original text via decoded_vocab_as_bytes.
• Some models pad their vocab size to a multiple of 32 or similar. If your model’s vocab size differs from encoded_vocab.len(), use new_with_vocab_size to pass the model’s vocab size so bitmask sizes are computed correctly.

Implementations

impl TokenizerInfo

pub fn new<T: AsRef<str>>( encoded_vocab: &[T], vocab_type: VocabType, stop_token_ids: &Option<Box<[i32]>>, add_prefix_space: bool, ) -> Self

Construct a TokenizerInfo with vocab size derived from encoded_vocab.

If the model’s vocab size differs from encoded_vocab.len(), prefer new_with_vocab_size.

pub fn new_with_vocab_size<T: AsRef<str>>( encoded_vocab: &[T], vocab_type: VocabType, vocab_size: Option<usize>, stop_token_ids: &Option<Box<[i32]>>, add_prefix_space: bool, ) -> Self

Construct a TokenizerInfo with an explicit model vocab_size.

Use this when the model’s vocab size (e.g., padded to a multiple of 32) differs from the tokenizer’s encoded_vocab.len(). Indices in the range [encoded_vocab.len(), vocab_size) are treated as special/reserved.

pub fn from_vocab_and_metadata_bytes<I, B>( encoded_vocab: I, metadata: &str, ) -> Self

where I: IntoIterator<Item = B>, B: AsRef<[u8]>,

Construct TokenizerInfo from encoded vocab (bytes) and a metadata JSON string produced by dump_metadata.

pub fn vocab_type(&self) -> VocabType

The type of the vocabulary.

pub fn vocab_size(&self) -> usize

The size of the vocabulary.

pub fn add_prefix_space(&self) -> bool

Whether the tokenizer will prepend a space before the text in the tokenization process.

pub fn decoded_vocab(&self) -> Box<[Box<[u8]>]>

The decoded vocabulary of the tokenizer. This converts tokens in the LLM’s vocabulary back to the original text form (e.g., ByteFallback “<0x1B>” -> “\u001b”).

pub fn stop_token_ids(&self) -> Box<[i32]>

Stop token ids.

pub fn special_token_ids(&self) -> Box<[i32]>

The special token ids. Special tokens include control tokens, reserved tokens, padded tokens, etc. Now it is automatically detected from the vocabulary.

pub fn dump_metadata(&self) -> String

Dump the metadata of the tokenizer to a json string. It can be used to construct the tokenizer info from the vocabulary and the metadata string.

pub fn serialize_json(&self) -> String

Serialize the tokenizer info to a JSON string.

pub fn deserialize_json(json: &str) -> Result<Self, String>

Deserialize a TokenizerInfo from a JSON string.

Returns

• Ok(TokenizerInfo) on success
• Err(String) when deserialization fails due to any of the following:
  • invalid JSON syntax
  • schema/format mismatch with TokenizerInfo serialization
  • serialization version mismatch (via the __VERSION__ field) The error string mirrors the C++ exception message.

Trait Implementations

impl Drop for TokenizerInfo

fn drop(&mut self)

Executes the destructor for this type.