Module tokenizer 

Tokenizers: the GPT-2 byte-level BPE, and nanoGPT’s character-level vocab.

Gpt2Tokenizer loads the original vocab.json + merges.txt. The split pattern uses a negative lookahead, which the standard regex crate cannot express, so fancy-regex is used (see roadmap “Known Pitfalls”).

The generation path in crate::Gpt2 is generic over the Tokenizer trait, so CharTokenizer plugs into it unchanged.

Re-exports

pub use self::char::CharTokenizer;

Modules

char

Character-level tokenizer, matching nanoGPT’s shakespeare_char prepare step: the vocabulary is the sorted set of distinct characters in the corpus, and a token id is that character’s index in the sorted order.

Structs

Gpt2Tokenizer

Enums

AnyTokenizer

Either tokenizer, for callers that pick one at runtime (the TUI and the wasm facade both do). Generation stays generic over impl Tokenizer, so this dispatches without a vtable.

Traits

Tokenizer

The tokenizer surface the generation path needs — exactly the four methods crate::Gpt2::generate and friends call.