Expand description
Byte-level BPE tokenizer — HF tokenizer.json parity.
Faithful pipeline (matches tokenizers for Qwen-style files):
added-token split (raw text) → NFC → pre-tokenizer regex
(GPT-2 style, needs lookahead) → byte-level mapping → ranked BPE
merges → vocab ids. Decode reverses through the byte-level map,
assembling UTF-8 across token boundaries.
No silent corruption: a symbol that cannot be encoded is reported (tracing::error), never dropped without a trace.
Structs§
- Tokenizer
- A loaded BPE tokenizer.