Module tokenizer 

A real, reversible byte-level tokenizer: each UTF-8 byte maps to token id byte as u32 (vocabulary 0..256). This is not a full BPE/tokenizer.json implementation – GLM-5.2, DeepSeek V4 Pro, and Kimi K3 each ship their own trained BPE vocabulary alongside their weights, and none of those vocab files are guessable or available in this environment (see docs/MODELS.md) – but unlike the previous placeholder (byte % vocab_size, which was lossy and could not decode back to the original text), this tokenizer is exact and round-trips perfectly. It is the honest “smallest real thing that works” rather than a fake stand-in.

Loading a real BPE merge table from a GGUF file’s tokenizer.ggml.tokens / tokenizer.ggml.merges metadata arrays (see ferrox-gguf’s GgufValue::Array support, already verified against a real downloaded llama.cpp vocab fixture) was the natural next step and now exists below (GgufBpeTokenizer, GgufSpmTokenizer, GgufUnigramTokenizer).

Structs

ByteTokenizer

GgufBpeTokenizer

A real BPE tokenizer built from a GGUF file’s own tokenizer.ggml.tokens / tokenizer.ggml.merges metadata arrays. Supports GPT-2 byte-remap BPE (tokenizer.ggml.model == "gpt2") and Gemma-4 SPM-style BPE ("gemma4": escape spaces to ▁, merge on raw UTF-8, newline-only pre-split).

GgufSpmTokenizer

A real SentencePiece-BPE tokenizer, built from a GGUF file’s tokenizer.ggml.tokens + tokenizer.ggml.scores metadata (tokenizer.ggml.model == "llama" in GGUF’s convention – this is SentencePiece’s BPE model type, not its Unigram model type, despite both living under the umbrella term “SentencePiece”; the distinction matters because the encode algorithms are different).

GgufUnigramTokenizer

A real SentencePiece Unigram (ULM) tokenizer, built from a GGUF file’s tokenizer.ggml.tokens + tokenizer.ggml.scores metadata (tokenizer.ggml.model == "t5" in GGUF’s convention – confirmed directly against llama.cpp’s real vocab-type-loading source (src/llama-vocab.cpp’s tokenizer_model == "t5" case), not guessed; T5-family models are the real-world users of this tag).

StopTokens

The set of token ids a decode loop must stop on, carried as one value so a caller cannot accidentally carry only half of it.

Enums

TokenizerLoadError

Functions

eog_token_ids

Every token id that ends generation, not just eos_token_id.

prepend_bos

Prepends the checkpoint’s BOS id to an already-encoded prompt, unless the prompt already starts with it.

should_add_bos_token

Whether prompt encoding should prepend the GGUF BOS token.