Struct StreamingDetokenizer 

pub struct StreamingDetokenizer<'t> { /* private fields */ }

Streaming detokenizer over a borrowed TokenizerHandle.

let mut detok = StreamingDetokenizer::new(&tok, true);
for id in decoded_token_stream {
    let delta = detok.push(id)?;   // emit to the client as it stabilizes
    if !delta.is_empty() { send(delta); }
}
send(detok.finish()?);             // flush any held-back tail

Implementations

impl<'t> StreamingDetokenizer<'t>

pub fn new(tok: &'t TokenizerHandle, skip_special: bool) -> Self

pub fn push(&mut self, id: u32) -> Result<String>

Append a token and return any newly-stable text (may be empty when the token only extends an incomplete multi-byte char).

pub fn finish(&mut self) -> Result<String>

Flush everything decoded so far, including a held-back tail. Call once at the end of generation (after the last push).

pub fn text(&self) -> Result<String>

The full text decoded so far (including any not-yet-emitted tail).

pub fn ids(&self) -> &[u32]

Tokens accumulated so far.