Expand description
memra-tokenizer — host-only GPT-2/BPE tokenizer (encode + decode + chat template).
Algorithm TAKEn ~1:1 from llama.cpp’s GPT-2 BPE path (src/llama-vocab.cpp,
src/unicode.cpp), Rust glue hand-rolled. Built from the model’s own GGUF
tokenizer metadata (tokenizer.ggml.*) so it is integer-exact for that model.
Scope: the gpt2 vocab model with the qwen35 pre-tokenizer (Qwen3.5). Other
pre-tokenizers are not ported (we only need this model’s).
Re-exports§
pub use chat::apply_chat_template_str;
Modules§
- chat
- Minimal chat-template renderer for the Qwen3.5 / ChatML format.