hermes-tokenizer

hermes-tokenizer is Hermes' stable-Rust byte-level BPE tokenizer. It loads a local Hugging Face tokenizer.json, keeps the artifact's token IDs unchanged, and exposes the operations used by training, inference, and model visualization:

let tokenizer = hermes_tokenizer::Tokenizer::from_file("tokenizer.json")?;
let ids = tokenizer.encode("A retrieval query", false)?;
let text = tokenizer.decode(&ids, true)?;

The current compatibility contract is intentionally narrow: byte-level BPE, NFC normalization, supported GigaToken split-regex families, Hugging Face added-token whitespace behavior, and the single-sequence parts of ByteLevel or TemplateProcessing post-processors. Unsupported model and pipeline types fail at load time instead of silently producing different token IDs.

The optimized merge engine and pretokenizers are extracted from GigaToken and run without Python, networking, Arrow, SentencePiece, or nightly Rust. See UPSTREAM.md for exact provenance and refresh policy.

Development

Run the unit and Hugging Face parity suites together:

cargo test -p hermes-tokenizer
cargo clippy -p hermes-tokenizer --all-targets -- -D warnings

Pretokenizer refactors must retain both whole-input and resumed/chunked boundaries. Keep scalar and SIMD family differences explicit and extend the parity fixtures before consolidating a hot-path policy.