Struct GgufBpeTokenizer 

pub struct GgufBpeTokenizer { /* private fields */ }

A real BPE tokenizer built from a GGUF file’s own tokenizer.ggml.tokens / tokenizer.ggml.merges metadata arrays. Supports GPT-2 byte-remap BPE (tokenizer.ggml.model == "gpt2") and Gemma-4 SPM-style BPE ("gemma4": escape spaces to ▁, merge on raw UTF-8, newline-only pre-split).

Verified against tests/fixtures/llama-bpe-vocab.gguf (GPT-2 path). See crates/ferrox-models/tests/gguf_vocab.rs.

Implementations

impl GgufBpeTokenizer

pub fn from_gguf(file: &impl TensorSource) -> Result<Self, TokenizerLoadError>

Loads the vocabulary + merge table from a GGUF file’s metadata. Merges are optional (some tokenizer types, e.g. byte-level unigram, don’t use them); if absent, encoding falls back to per-byte token lookup. tokenizer.ggml.model == "gemma4" selects SPM-whitespace BPE; everything else with merges uses GPT-2 style.

pub fn vocab_size(&self) -> usize

pub fn has_merges(&self) -> bool

pub fn encode_word(&self, word: &str) -> Vec<u32>

Greedy BPE merge over one pre-split chunk. GPT-2 style remaps bytes through byte_to_unicode; Gemma-4 style merges raw UTF-8 codepoints (after " " → ▁ escaping in encode).

pub fn encode(&self, text: &str) -> Vec<u32>

Encodes text: specials first, then style-specific pretokenize + encode_word. Gemma-4 escapes spaces to ▁ and splits only on newlines; newline-only chunks look up the whole string in vocab (multi-newline tokens) before BPE.

pub fn decode(&self, ids: &[u32]) -> String

GPT-2: remapped unicode → bytes. Gemma-4: unescape ▁ → space and expand <0xXX> byte tokens (same shape as SPM decode).