Struct GgufUnigramTokenizer 

pub struct GgufUnigramTokenizer { /* private fields */ }

A real SentencePiece Unigram (ULM) tokenizer, built from a GGUF file’s tokenizer.ggml.tokens + tokenizer.ggml.scores metadata (tokenizer.ggml.model == "t5" in GGUF’s convention – confirmed directly against llama.cpp’s real vocab-type-loading source (src/llama-vocab.cpp’s tokenizer_model == "t5" case), not guessed; T5-family models are the real-world users of this tag).

How this differs from GgufSpmTokenizer

Both are “SentencePiece” vocabularies, but with entirely different encoding algorithms: GgufSpmTokenizer implements SentencePiece’s BPE model type (a merge-rank table, greedy pairwise merging). Unigram has no merge table at all – every vocabulary entry carries a real log-probability score, and the optimal (highest total log-probability) segmentation of the whole input is found by a forward Viterbi dynamic-programming pass: best[j] is the highest- scoring way to reach position j, computed as max over every vocabulary piece P that ends at j of best[j - len(P)] + score(P). This is reimplemented independently against real llama.cpp source read for this purpose (src/llama-vocab.cpp’s llm_tokenizer_ugm_session class) – not copied, but the algorithm (including its unknown-token fallback score and tie-breaking) is transcribed deliberately rather than guessed, since a plausible-looking-but-wrong Viterbi variant would silently produce different segmentations than the model was actually trained to expect.

Preprocessing matches GgufSpmTokenizer’s exactly (' ' -> ▁ U+2581, plus a leading ▁) – both are real SentencePiece conventions, this being the default add_dummy_prefix=true / treat_whitespace_as_suffix=false behavior. Real SentencePiece models can optionally ship a precompiled_charsmap (an auxiliary normalization table, e.g. NFKC folding) via GGUF’s tokenizer.ggml.precompiled_charsmap key; this implementation does not read or apply it (a real, disclosed scope decision, not an oversight – llama.cpp’s own loader treats this key as optional too, falling back to plain UTF-8 handling when absent).

Unlike GgufSpmTokenizer, Unigram has no byte-fallback token convention in the real reference implementation: a character with no matching vocabulary entry is scored via a fixed unknown-token penalty (min_score - 10.0, matching the real unknown_token_score_penalty constant) and mapped to the vocabulary’s real unknown-token id (tokenizer.ggml.unknown_token_id, defaulting to 0 if absent) rather than expanded into raw bytes.

Real user-defined/control tokens (GGUF’s tokenizer.ggml.token_type metadata) are not yet given longest-match priority over the Viterbi pass the way the real reference implementation does – deferred alongside GgufSpmTokenizer’s equivalent gap (chat-template special-token handling), rather than solved once per tokenizer independently.

Verification

Cross-validated against a real Unigram model trained with the real sentencepiece Python library (not a hand-built fixture) – exact-match token-id-sequence comparison across ASCII text, mixed-case, punctuation, digit runs, repeated whitespace, and non-ASCII (accented Latin) text, plus text containing no matching vocabulary substrings at all (exercising the unknown-token fallback repeatedly).

Implementations

impl GgufUnigramTokenizer

pub fn from_gguf(file: &impl TensorSource) -> Result<Self, TokenizerLoadError>

pub fn vocab_size(&self) -> usize

pub fn encode(&self, text: &str) -> Vec<u32>

Encodes text via the real forward-Viterbi Unigram algorithm described in this struct’s doc comment. Score accumulation uses f64 (matching the real reference’s double score_sum), since summing many f32 log-probabilities over a long input can accumulate enough rounding error to flip which of two near-tied segmentations looks best.

Control/user-defined tokens (chat-template markers) are first carved out as atomic substrings via split_on_special_tokens; each remaining raw-text run is Viterbi-segmented independently.

pub fn decode(&self, ids: &[u32]) -> String

Reverses encode’s ' ' <-> ▁ convention. Unigram has no byte-fallback token convention (see this struct’s doc comment), so every token here is decoded as plain text.