Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Implementations

impl Tokenizer

pub fn from_gguf(g: &GgufFile) -> Result<Self, String>

Build a tokenizer from a model’s GGUF tokenizer metadata.

pub fn from_hf_dir(dir: &Path) -> Result<Self, String>

Build a tokenizer from an HF fast-tokenizer checkpoint directory (tokenizer.json + optional tokenizer_config.json / generation_config.json / chat_template.jinja). Only byte-level BPE (the gpt2 class — MiniMax-M3, Qwen, Llama-3 style) is supported: model.type == "BPE" with a ByteLevel pre-tokenizer.

Mapping to the GGUF-built struct:

• model.vocab (token -> id map) -> id_to_token / token_to_id
• model.merges (“a b” strings OR [a,b] pairs; both HF serializations) -> bpe_ranks
• added_tokens special=true -> Control class (split before BPE + hidden on decode); non-special added tokens stay Normal.
• eos/bos: tokenizer_config eos_token/bos_token (string or {content} object), generation_config eos_token_id (int or array) as the eos fallback.
• add_bos: tokenizer_config add_bos_token (default false).
• chat template: tokenizer_config chat_template, else chat_template.jinja.
• tokenizer_config.pretokenize_regex equal to Qwen’s shipped regex -> qwen35
• unknown regex/missing metadata -> default, which warns before the fallback split

pub fn eos_id(&self) -> u32

pub fn eog_ids(&self) -> Vec<u32>

End-of-generation ids: eos + the common turn-end control tokens present in the vocab (llama’s special_eog set — <|im_end|> chatml, <turn|>/<end_of_turn> gemma).

pub fn bos_id(&self) -> Option<u32>

pub fn vocab_size(&self) -> usize

pub fn pre(&self) -> &str

pub fn chat_template(&self) -> Option<&str>

pub fn encode(&self, text: &str, add_special: bool) -> Vec<u32>

Encode text -> token ids.

add_special controls whether a BOS is prepended when the model asks for it. parse_special (always true here) splits control/user-defined/unknown tokens (e.g. <|im_start|>) out before BPE — matching llama’s default tokenize().

pub fn encode_special( &self, text: &str, add_special: bool, parse_special: bool, ) -> Vec<u32>

pub fn decode(&self, ids: &[u32]) -> String

Decode token ids -> String. special=false drops control tokens (chat tags); special=true renders them as their literal text.

pub fn token_is_control(&self, id: u32) -> bool

True for Control/Unknown tokens — vocab entries that are protocol markers, not text. External vocab consumers (llguidance’s toktrie, constrained decoding) must not let a grammar match these as literal bytes (a JSON string could otherwise smuggle <|im_start|>); they substitute a non-text marker form instead.

pub fn decode_special(&self, ids: &[u32], special: bool) -> String

pub fn decode_bytes_special(&self, ids: &[u32], special: bool) -> Vec<u8>

Decode token ids to their exact byte stream. Streaming callers must retain incomplete UTF-8 suffixes across token boundaries instead of replacing them prematurely.

pub fn apply_chat_template( &self, messages: &[(&str, &str)], add_generation_prompt: bool, ) -> String

Apply the chat template (from GGUF, or a chatml fallback) to a list of (role, content) turns, producing the prompt string. Then encode it.

pub fn apply_chat_template_tools( &self, turns: &[Turn], add_generation_prompt: bool, tools_json: &[String], think: ThinkMode, reasoning_effort: Option<&str>, ) -> Result<String, String>

Tools-capable chat rendering (OpenAI tools / tool_calls / role:“tool” surface + the think-tail switch + the step35 reasoning_effort string). Plain requests render byte-identically to apply_chat_template; see chat::apply_chat_template_tools.