Crate kopitiam_tokenizer 

Kopitiam Runtime: byte-level BPE tokenizer.

This crate turns text into the token ids a GPT-2/GPT-3/GPT-4- or Qwen-family model was trained on, and back. It is a from-scratch, original Rust implementation – not a wrapper around the tokenizers crate – because the whole point of the Kopitiam Runtime (see docs/ai-decisions/AID-0001) is to own this layer rather than depend on it, in keeping with this workspace’s Pure Rust Core commitment.

Byte-level BPE, in one paragraph

Classic word-level BPE needs an <UNK> token for anything outside its training vocabulary – a stray emoji, a typo, a byte sequence that isn’t valid text at all. Byte-level BPE sidesteps this entirely by making the 256 possible byte values the base alphabet instead of characters or words: every possible input, valid UTF-8 or not, is some sequence of bytes, and every byte already has a token id. There is therefore no <UNK> in this crate’s design and no failure mode for encoding – only Tokenizer::decode can fail, and only on a genuinely unknown id (see its docs). This is the scheme GPT-2 through GPT-4 and the Qwen family all use.

How the pieces fit together

• byte_map – the reversible byte <-> “printable Unicode character” mapping that lets a byte-level vocab be written as JSON text at all. Used only when loading/saving tokenizer.json; the runtime path never touches it.
• vocab – vocab::Vocab, the token (raw bytes) <-> id table.
• merges – merges::MergeTable, the ordered “which adjacent pair merges first, into what” rules learned during BPE training.
• pretokenize – splits text into words/numbers/punctuation/ whitespace-run chunks before BPE runs, matching the GPT-2 pattern without regex’s unsupported lookahead (see that module’s docs – this is the single easiest place to introduce a silent correctness bug in a tokenizer).
• specials – atomic special-token matching (<|endoftext|> and friends), so they are pulled out before pre-tokenization ever sees them.
• bpe – bpe::BpeTokenizer, the concrete Tokenizer that wires the above into encode/decode.
• loader – parses the HuggingFace tokenizer.json shape into a bpe::BpeTokenizer.
• estimate – a dependency-free, deterministic token count estimator (estimate::estimate_tokens) for when there is no loaded vocab to run the real BPE against. It exists to make LLM cost visible (read-vs-outline) rather than to reproduce exact ids.

What this crate does not do

No model inference, no chat templating, no attention-mask bookkeeping. Those belong to kopitiam-runtime, which will code against the Tokenizer trait rather than BpeTokenizer directly, the same way it will code against traits from kopitiam-tensor and kopitiam-loader rather than their concrete types.

Re-exports

pub use bpe::BpeTokenizer;

pub use estimate::LineEstimate;

pub use estimate::estimate_tokens;

pub use estimate::estimate_tokens_by_line;

pub use loader::from_tokenizer_json;

Modules

bpe

BpeTokenizer: the byte-level BPE crate::Tokenizer implementation.

byte_map

The GPT-2 “byte-to-unicode” alphabet.

estimate

Dependency-free, deterministic token count estimation – the “make cost visible” primitive from the token-max plan (§0.7): let an agent choose read-vs-outline informed by an approximate token cost rather than blind.

loader

Loading a HuggingFace tokenizer.json.

merges

The ordered BPE merge rules.

pretokenize

GPT-2/Qwen-style pre-tokenization: splitting text into chunks (words, numbers, punctuation runs, contractions, whitespace runs) before BPE merges ever run.

specials

Special tokens (<|endoftext|>, <|im_start|>, <|im_end|>, …) and atomic matching.

vocab

The token <-> id table shared by encode and decode.

Traits

Tokenizer

What every tokenizer implementation in the Kopitiam Runtime must provide. kopitiam-runtime codes against this trait, not BpeTokenizer directly, so a future tokenizer scheme (e.g. SentencePiece for a model family that needs it) can be dropped in without touching the runtime’s call sites.