Skip to main content

Module traits

Module traits 

Source

Enums§

DecodeResult
Result of decoding token IDs to text.

Traits§

Decoder
Implementations must ensure that partial multi-byte sequences produce U+FFFD (\u{FFFD}) in the output rather than returning Err. This is commonly achieved via String::from_utf8_lossy (tiktoken) or library-internal byte-fallback handling (HuggingFace). DecodeStream::step() relies on DecodeResult::Partial to detect incomplete sequences and buffer tokens until the full character arrives.
Encoder
Tokenizer