Expand description
Stable-Rust, byte-level BPE tokenization for Hermes.
The optimized merge engine and pretokenizers are derived from GigaToken
0.10.0. The public wrapper is intentionally narrower: local or in-memory
Hugging Face tokenizer.json artifacts, persistent per-thread caches,
deterministic batch order, and the vocabulary operations required by
Hermes training, inference, and visualization.
Structs§
- Tokenizer
- Thread-safe tokenizer with persistent single-document and Rayon-worker
caches. Token IDs remain those of the source
tokenizer.json.
Constants§
- UPSTREAM_
GIGATOKEN_ REVISION - GigaToken revision from which the optimized byte-level core was extracted.