Struct TokenStream 

pub struct TokenStream {
    pub flush_at_whitespace: bool,
    /* private fields */
}

Byte-level detokenizer buffer with partial-token accumulation.

Raw model output often arrives as byte slices that do not align with UTF-8 character boundaries (e.g. multi-byte CJK characters split across two model tokens). TokenStream accumulates bytes until a complete UTF-8 sequence is available, then returns the decoded string.

Fields

flush_at_whitespace: bool

If true, the stream defers flushing until a whitespace boundary is found. This can be useful for word-level de-tokenization.

Implementations

impl TokenStream

pub fn new() -> Self

Create a new empty TokenStream.

pub fn push_token_bytes(&mut self, bytes: &[u8]) -> Option<String>

Append bytes to the internal buffer.

Returns Some(text) if the buffer now forms a valid complete UTF-8 string, or None if more bytes are still needed to complete a multi-byte character.

pub fn flush(&mut self) -> String

Force-flush whatever remains in the buffer as lossy UTF-8.

Any invalid byte sequences are replaced with U+FFFD (replacement char).

pub fn is_empty(&self) -> bool

Returns true if the internal buffer is empty.

Trait Implementations

impl Default for TokenStream

fn default() -> Self

Returns the “default value” for a type.