Expand description
Integration of the HuggingFace tokenizers library with toktrie,
providing byte-level tokenization support. This crate wraps HuggingFace
tokenizers and adapts them for use with the toktrie token trie
infrastructure.
Structsยง
- Byte
Tokenizer - A HuggingFace tokenizer adapted for byte-level token processing.
- Byte
Tokenizer Env - Combines a
ByteTokenizerwith aTokTrieand implements theTokenizerEnvtrait.