pub fn get_encoding(name: &str) -> Option<&'static CoreBpe>Expand description
Get a cached tokenizer by encoding name.
Supported encodings:
- OpenAI:
cl100k_base,o200k_base,o200k_harmony,p50k_base,p50k_edit,r50k_base,gpt2 - Meta:
llama3 - DeepSeek:
deepseek_v3,deepseek_v4 - Alibaba:
qwen2 - Mistral:
mistral_v3 - Moonshot:
kimi_k2,kimi_k3 - Zhipu:
glm4,glm5 - MiniMax:
minimax_m2
Note: gpt2 is a name-level alias for r50k_base; both return the same cached instance.