hermes-tokenizer

hermes-tokenizer is Hermes' stable-Rust byte-level BPE tokenizer. It loads a local Hugging Face tokenizer.json, keeps the artifact's token IDs unchanged, and exposes the operations used by training, inference, and model visualization:

let tokenizer = hermes_tokenizer::Tokenizer::from_file("tokenizer.json")?;
let ids = tokenizer.encode("A retrieval query", false)?;
let text = tokenizer.decode(&ids, true)?;

The current compatibility contract is intentionally narrow: byte-level BPE, NFC normalization, supported GigaToken split-regex families, Hugging Face added-token whitespace behavior, and the single-sequence parts of ByteLevel or TemplateProcessing post-processors. Unsupported model and pipeline types fail at load time instead of silently producing different token IDs.

The optimized merge engine and pretokenizers are extracted from GigaToken and run without Python, networking, Arrow, SentencePiece, or nightly Rust. See UPSTREAM.md for exact provenance and refresh policy.