Struct DecodeStreamState 

pub struct DecodeStreamState { /* private fields */ }

Per-stream UTF-8-safe decode state. Owned by the caller.

BPE / byte-level tokenizers (Qwen3, GPT-2, etc.) sometimes emit a single token that carries only part of a multi-byte UTF-8 character (e.g. one byte of a CJK ideograph or emoji). Decoding tokens one-at-a-time without buffering breaks those multi-byte sequences and produces U+FFFD replacement characters in the output stream. This state mirrors what the HuggingFace tokenizers::DecodeStream keeps internally so that we can own it externally and feed tokens through TokenizerBridge::step_decode.

Use one DecodeStreamState per generation request; reset (or drop & re-create) it between independent requests.

Implementations

impl DecodeStreamState

pub fn new(skip_special_tokens: bool) -> Self

Construct a fresh decode-stream state.

skip_special_tokens matches the existing decode() behavior — pass true to drop sentinel tokens (e.g. <|im_end|>) from the output.

pub fn reset(&mut self)

Reset the state, preserving the original skip_special_tokens flag.

Trait Implementations

impl Default for DecodeStreamState

fn default() -> DecodeStreamState

Returns the “default value” for a type.