Struct GgufWordPieceTokenizer 

pub struct GgufWordPieceTokenizer { /* private fields */ }

A real WordPiece tokenizer built from a GGUF file’s own tokenizer.ggml.tokens metadata array.

See the module docs for the algorithm and for why the vocabulary looks the way it does. Checked against llama.cpp on the same GGUF by crates/ferrox-models/tests/wordpiece_parity.rs.

Implementations

impl GgufWordPieceTokenizer

pub fn from_gguf(file: &impl TensorSource) -> Result<Self, TokenizerLoadError>

pub fn vocab_size(&self) -> usize

pub fn normalizer(&self) -> NormalizerOptions

pub fn encode(&self, text: &str) -> Vec<u32>

Encodes text, with no [CLS]/[SEP] added.

Upstream’s add_special wraps the result in BOS and SEP; that decision lives with super::should_add_bos_token and super::prepend_bos for every tokenizer in this crate, and baking it in here would double it for callers that already do it.

pub fn decode(&self, ids: &[u32]) -> String

The text a token id stands for.

Reverses the converter’s phantom space, so ▁hello decodes to " hello" and the continuation ing decodes to "ing". Concatenating a sequence therefore reproduces the normalized text with a leading space, which is what llama.cpp’s llama_unescape_whitespace produces before its clean_spaces pass trims the first one.

A WordPiece round trip is lossy no matter what this does: the normalizer lowercased, stripped accents and dropped controls before any of these ids existed. Round-tripping is therefore NOT evidence that this tokenizer is right, which is why the tests that matter compare ids against llama.cpp instead.