memra-tokenizer 0.81.1

GGUF-native BPE/SPM tokenizer with chat-template support for the memra inference engine
Documentation

memra-tokenizer — host-only GPT-2/BPE tokenizer (encode + decode + chat template).

Algorithm TAKEn ~1:1 from llama.cpp's GPT-2 BPE path (src/llama-vocab.cpp, src/unicode.cpp), Rust glue hand-rolled. Built from the model's own GGUF tokenizer metadata (tokenizer.ggml.*) so it is integer-exact for that model.

Scope: the gpt2 vocab model with the qwen35 pre-tokenizer (Qwen3.5). Other pre-tokenizers are not ported (we only need this model's).