Expand description
Backend-neutral tokenizer construction and chat-template rendering.
This crate adapts Hugging Face tokenizers and Jinja chat templates for language-model runtimes. It can reconstruct tokenizers from GGUF metadata, import supported tiktoken vocabularies, and render structured conversations without depending on a model execution backend.
Modulesยง
- error
- Error types returned by tokenizer and template operations.
- gguf
- Tokenizer reconstruction from GGUF metadata. Reconstruction of executable tokenizers from portable GGUF metadata.
- tiktoken
- Importers for rank-ordered tiktoken vocabularies. Import rank-ordered tiktoken byte-BPE files into a fast tokenizer.
- tokenizer
- Tokenizer wrappers and chat-template rendering utilities.