Skip to main content

Module tokenizer

Module tokenizer 

Source
Expand description

§Tokenizer

Provides lightweight, pure-Rust tokenization and detokenization routines with byte-fallback support and subword vocabulary mapping.

Structs§

ByteLevelTokenizer
A robust byte-level tokenizer with subword vocabulary lookup and lossless byte-fallback.

Traits§

Tokenizer
Standard abstract interface for tokenizers in spec-stream.