Struct Detokenizer 

pub struct Detokenizer { /* private fields */ }

Stateful detokenizer.

Construct with a TokenizerMap; call Detokenizer::render repeatedly with chunks of IDs. State (the partial UTF-8 byte buffer) persists across calls; Detokenizer::reset clears it.

Implementations

impl Detokenizer

pub fn new(map: &TokenizerMap) -> Self

Build a detokenizer from a map.

pub fn render(&mut self, ids: &[u32], options: DetokenizeOptions) -> String

Render a chunk of IDs to text. Stateful across calls.

pub fn reset(&mut self)

Reset internal state — call between conversations / requests.

pub fn detokenize( map: &TokenizerMap, ids: &[u32], render_special: bool, ) -> String

Convenience: detokenize a complete sequence in one shot. Uses a fresh detokenizer; partial buffering not exposed.