Expand description
§Tokenizer
Provides lightweight, pure-Rust tokenization and detokenization routines with byte-fallback support and subword vocabulary mapping.
Structs§
- Byte
Level Tokenizer - A robust byte-level tokenizer with subword vocabulary lookup and lossless byte-fallback.
Traits§
- Tokenizer
- Standard abstract interface for tokenizers in
spec-stream.