Crate memra_tokenizer 

memra-tokenizer — host-only GPT-2/BPE tokenizer (encode + decode + chat template).

Algorithm TAKEn ~1:1 from llama.cpp’s GPT-2 BPE path (src/llama-vocab.cpp, src/unicode.cpp), Rust glue hand-rolled. Built from the model’s own GGUF tokenizer metadata (tokenizer.ggml.*) so it is integer-exact for that model.

Scope: the gpt2 vocab model with the qwen35/qwen2/deepseek-v3 pre-tokenizers, plus the gemma4 SPM-style path — see SUPPORTED_PRETOKENIZERS. A model declaring anything else is REFUSED at load (UnknownPretokenizer), because an unported pre-tokenizer produces fluent output with wrong token ids and nothing downstream can see it.

Re-exports

pub use chat::apply_chat_template_str;

Modules

chat

Minimal chat-template renderer for the Qwen3.5 / ChatML format.

Structs

Tokenizer

UnknownPretokenizer

A model declared a pre-tokenizer memra has no exact split for.

Enums

PreSplit

The pre-tokenizer split a loaded Tokenizer runs. Constructed only through PreSplit::resolve, so “we do not know how to split for this model” is not a state a live tokenizer can be in unless the operator asked for it via the env opt-out.

Constants

ALLOW_UNKNOWN_PRETOKENIZER_ENV

Escape hatch for deliberate experimentation with a family whose pre-tokenizer is not ported yet. Set to 1 to downgrade the hard load error to a loud per-load WARN.

SUPPORTED_PRETOKENIZERS

Every tokenizer.ggml.pre id memra implements an EXACT split for. This is the allowlist a load is checked against and the list quoted in the load error, so the two can never drift.