Struct GgufSpmTokenizer 

pub struct GgufSpmTokenizer { /* private fields */ }

A real SentencePiece-BPE tokenizer, built from a GGUF file’s tokenizer.ggml.tokens + tokenizer.ggml.scores metadata (tokenizer.ggml.model == "llama" in GGUF’s convention – this is SentencePiece’s BPE model type, not its Unigram model type, despite both living under the umbrella term “SentencePiece”; the distinction matters because the encode algorithms are different).

How this differs from GgufBpeTokenizer

GgufBpeTokenizer implements GPT2-style BPE: a fixed merge-rank table applied greedily left-to-right after GPT2’s own byte-to-unicode remap and regex pre-tokenization. SentencePiece-BPE vocabularies (used by the original LLaMA, and generally any model whose GGUF reports tokenizer.ggml.model = "llama") don’t ship a merge-rank table at all – instead every vocabulary entry carries a score, and encoding works by repeatedly merging whichever currently adjacent pair of symbols forms the highest-scoring known vocabulary piece, using a priority queue over merge candidates (this is the llm_tokenizer_spm algorithm from llama.cpp, reimplemented here independently against the public GGUF metadata, not from llama.cpp source). Preprocessing replaces spaces with ▁ (U+2581) and adds a leading ▁, matching SentencePiece’s own convention, rather than GPT2’s byte-to-unicode remap.

A real bug found and fixed while building this

The first implementation of this algorithm checked merge-candidate validity by adjacency alone (is this pair still directly next to each other in the linked list?). That’s necessary but not sufficient: a symbol’s content can change between when a candidate merge is queued and when it’s popped, if that symbol was itself the survivor of a different merge in the meantime, while staying adjacency-valid at the same list position. The fix is to also store the exact left/right text expected at queue time and re-check it at pop time, discarding (not re-queuing) any candidate whose content has since changed. This was caught immediately by testing against real reference data (see below) rather than by code review – the bug produced plausible-looking but wrong output (“Hello world” tokenized as 6 pieces instead of the correct 2) which would have been easy to miss without a real ground truth to check against.

Verification

Tested against tests/fixtures/llama-spm-vocab.gguf (downloaded directly from ggml-org/llama.cpp’s own repository, the real LLaMA-1/2 tokenizer vocabulary) and its accompanying .gguf.inp/.gguf.out files – llama.cpp’s own CI test corpus of 45 input strings and their exact expected token ID sequences, covering ASCII, whitespace runs, control characters, CJK/Khmer/ Vietnamese text, emoji, and byte-fallback. All 45 match exactly.

Implementations

impl GgufSpmTokenizer

pub fn from_gguf(file: &impl TensorSource) -> Result<Self, TokenizerLoadError>

pub fn vocab_size(&self) -> usize

pub fn encode(&self, text: &str) -> Vec<u32>

Encodes text using SentencePiece’s space-replacement convention (' ' -> ▁, plus a leading ▁) and the score-prioritized pairwise-merge algorithm described in this struct’s doc comment. Characters with no direct vocabulary entry are expanded to UTF-8 byte-fallback tokens (<0xXX>, which every real SentencePiece-BPE vocabulary includes for exactly this purpose) before merging begins.

Control/user-defined tokens (chat-template markers like <|user|>) are first carved out as atomic substrings via split_on_special_tokens, matching real llama.cpp’s tokenizer_st_partition behavior, so they’re never shattered into byte-fallback pieces; each remaining raw-text run between them is merged independently. A leading dummy ▁ is applied to a run only when Self::add_space_prefix is true (llama.cpp add_space_prefix && is_prev_special for each fragment).

pub fn decode(&self, ids: &[u32]) -> String