Module detokenize 

Stateful detokenizer: token IDs → text.

Three correctness concerns it handles:

1. Per-token decoding via the map’s encoder (byte_level / metaspace / identity).
2. Byte-fallback range — IDs in [byte_fallback_start, byte_fallback_end] are decoded as raw bytes.
3. Partial multi-byte sequences across frame boundaries — buffered between calls when partial: true.

Structs

DetokenizeOptions

Options for Detokenizer::render.

Detokenizer

Stateful detokenizer.