Module kimi_tokenizer 

Kimi K3’s real tokenizer: tiktoken-style rank-based BPE, loaded from a real tiktoken.model file (base64-encoded byte sequence + rank per line – the standard OpenAI tiktoken vocab format, confirmed against a real downloaded Kimi K3 tiktoken.model and its real tokenization_kimi.py/tokenizer_config.json), not from GGUF metadata like GgufBpeTokenizer/GgufSpmTokenizer – Kimi K3 ships as safetensors, with its own real tokenizer format, distinct from both existing tokenizers in this module.

The real split pattern (pat_str in tokenization_kimi.py) uses Unicode script/general-category properties (\p{Han}, \p{Lu}, …) plus a negative lookahead (\s+(?!\S)) that Rust’s regex crate deliberately doesn’t support (no backtracking, for linear-time guarantees) – fancy-regex is used instead, since it supports exactly this class of pattern while still being a generic, model-agnostic regex engine (same category of tool as regex itself), not tiktoken-specific code. One real translation was needed: the real pattern uses [A&&[^B]] (character-class set intersection), valid in Python’s third-party regex module (which tiktoken’s reference implementation depends on) but not in Rust’s regex syntax. Rewritten as (?:(?!B)[A]) – a per-character negative lookahead guarding the class – which is semantically equivalent under repetition (*/+) since each repeated character is independently re-checked. Verified byte-for-byte against the real tiktoken Python library (see this module’s tests).

The core merge algorithm (given a UTF-8 text piece already isolated by the split regex, repeatedly merge the lowest-rank adjacent byte span until no mergeable pair remains) is transcribed from OpenAI’s publicly documented tiktoken algorithm description, not copied from any source file.

Structs

KimiTokenizer

Enums

KimiTokenizerError

Functions

parse_special_tokens

Parses the real added_tokens_decoder block of Kimi K3’s tokenizer_config.json ({"163584": {"content": "[BOS]", ...}, ...}, real key/value shapes confirmed against the real downloaded file) into a name -> id map suitable for KimiTokenizer::new’s special_tokens argument.

parse_tiktoken_vocab

Parses a real .tiktoken-format vocab file: one <base64-bytes> <rank> pair per line, blank lines ignored. Standard OpenAI tiktoken vocab format (confirmed against a real downloaded Kimi K3 tiktoken.model).