Struct ByteLevelStreamingDecoder 

pub struct ByteLevelStreamingDecoder<'a> { /* private fields */ }

A streaming decoder for ByteLevel-encoded tokenizers (GPT-2, Llama, DeepSeek V3).

This decoder handles the ByteLevel encoding used by some tokenizers where raw bytes are mapped to printable Unicode characters. It first decodes the ByteLevel representation back to raw bytes, then assembles them into valid UTF-8 strings.

Example

use splintr::{from_pretrained, Backend, ByteLevelStreamingDecoder};

let any = from_pretrained("deepseek_v3")?;
let Backend::Bpe(tokenizer) = any.into_backend() else {
    unreachable!("deepseek_v3 loads as a BPE backend");
};
let mut decoder = ByteLevelStreamingDecoder::new(&tokenizer);

for token_id in tokenizer.encode("Hello, world!") {
    if let Some(text) = decoder.add_token(token_id) {
        print!("{}", text);
    }
}
// Flush any remaining buffered bytes
print!("{}", decoder.flush());

Implementations

impl<'a> ByteLevelStreamingDecoder<'a>

pub fn new(tokenizer: &'a Tokenizer) -> Self

Create a new ByteLevel streaming decoder for the given tokenizer.

pub fn add_token(&mut self, token_id: u32) -> Option<String>

Add a token and return any complete UTF-8 characters.

The token’s ByteLevel-encoded bytes are first decoded to raw bytes, then assembled into valid UTF-8 strings.

Returns Some(string) if there are complete characters to emit, or None if the current bytes are still incomplete.

pub fn add_tokens(&mut self, token_ids: &[u32]) -> Option<String>

Add multiple tokens at once and return complete UTF-8 characters.

pub fn flush(&mut self) -> String

Flush any remaining buffered bytes.

If there are incomplete UTF-8 sequences in the buffer, they will be replaced with the Unicode replacement character (U+FFFD).

pub fn reset(&mut self)

Reset the decoder state, discarding any buffered bytes.

pub fn has_pending(&self) -> bool

Check if there are buffered bytes waiting for completion.

pub fn pending_bytes(&self) -> usize

Get the number of pending bytes in the buffer.