memra-tokenizer — host-only GPT-2/BPE tokenizer (encode + decode + chat template).
Algorithm TAKEn ~1:1 from llama.cpp's GPT-2 BPE path (src/llama-vocab.cpp,
src/unicode.cpp), Rust glue hand-rolled. Built from the model's own GGUF
tokenizer metadata (tokenizer.ggml.*) so it is integer-exact for that model.
Scope: the gpt2 vocab model with the qwen35/qwen2/deepseek-v3 pre-tokenizers, plus
the gemma4 SPM-style path — see SUPPORTED_PRETOKENIZERS. A model declaring anything else
is REFUSED at load (UnknownPretokenizer), because an unported pre-tokenizer produces
fluent output with wrong token ids and nothing downstream can see it.