Expand description
Tokenizer wrapper — encode text to token IDs and decode back.
Uses the HuggingFace tokenizers crate, loading from either
a tokenizer.json file (HF format) or extracting from GGUF metadata.
Structs§
- Tokenizer
- Wrapper around the HuggingFace tokenizer.
Enums§
- Tokenizer
Error - Errors during tokenizer operations.