Struct StreamingDecoder 

pub struct StreamingDecoder { /* private fields */ }

A streaming decoder that handles incomplete UTF-8 sequences across token boundaries.

When decoding tokens one at a time (as in streaming LLM output), a token’s bytes may end in the middle of a multi-byte UTF-8 character. This decoder buffers incomplete sequences and only returns complete, valid UTF-8 strings.

Obtaining one

There is no public constructor: a decoder is built by a tokenizer’s own streaming_decoder — Tokenizer::streaming_decoder, SpmTokenizer::streaming_decoder or SentencePieceTokenizer::streaming_decoder or WordPieceTokenizer::streaming_decoder — which takes the surfaces, the skipped-special-token set and every spelling rule (byte level, byte fallback, metaspace) from that tokenizer’s own configuration. A decoder therefore cannot be paired with the wrong kind of vocabulary — the mistake that used to turn a byte-level stream into mojibake is not expressible.

A tokenizer loaded from a tokenizer.json has one more factory, AnyTokenizer::streaming_decoder: it takes those same rules from the file’s declared decoder pipeline when one is declared, and delegates to the backend factory above when none is — the same choice AnyTokenizer::decode makes.

Agreement with whole-sequence decoding

For any ids, concatenating every emission plus the final flush equals Tokenizer::decode_lossy of the same ids, and equals Tokenizer::decode whenever that succeeds. (A stream cannot see the future, so bytes that are still invalid at flush become U+FFFD instead of an error — that is the one and only case where strict whole-sequence decoding reports something the stream renders lossily.) Chunk boundaries affect only when text is emitted, never what.

The decoder owns its state and borrows nothing, so it can be moved into a generation task and outlive the scope the tokenizer was created in.

Example

use splintr::{from_pretrained, Backend};

let any = from_pretrained("cl100k_base")?;
let Backend::Bpe(tokenizer) = any.into_backend() else {
    unreachable!("cl100k_base loads as a BPE backend");
};
let mut decoder = tokenizer.streaming_decoder();

for token_id in tokenizer.encode("Hello, world!") {
    if let Some(text) = decoder.add_token(token_id)? {
        print!("{}", text);
    }
}
// Flush any remaining buffered bytes
print!("{}", decoder.flush());

Implementations

impl StreamingDecoder

pub fn add_token(&mut self, id: u32) -> Result<Option<String>, TokenizeError>

Add a token and return any complete UTF-8 characters.

Returns Some(string) if there are complete characters to emit, or None if the current bytes are still incomplete.

Errors with TokenizeError::InvalidTokenId if the id is in neither the vocabulary nor the special tokens, matching Tokenizer::decode. Ids already accepted stay buffered; reset discards them.

pub fn add_tokens( &mut self, ids: &[u32], ) -> Result<Option<String>, TokenizeError>

Add multiple tokens at once and return complete UTF-8 characters.

Feeding ids in groups is indistinguishable from feeding them one by one: only the emission points differ, never the concatenated text.

pub fn add_token_lossy(&mut self, id: u32) -> Option<String>

Add a token, skipping it if it is in no table.

The permissive twin of add_token, matching Tokenizer::decode_lossy. This never fails, so on_unknown is instantiated with Infallible, letting the compiler prove the Err arm away rather than a runtime assertion claiming it.

pub fn add_tokens_lossy(&mut self, ids: &[u32]) -> Option<String>

Add multiple tokens at once, skipping any that are in no table.

pub fn flush(&mut self) -> String

Flush any remaining buffered bytes.

If there are incomplete UTF-8 sequences in the buffer, they will be replaced with the Unicode replacement character (U+FFFD).

pub fn reset(&mut self)

Reset the decoder state, discarding any buffered bytes.

The decoder is then indistinguishable from a freshly built one.

pub fn has_pending(&self) -> bool

Check if there are buffered bytes waiting for completion.

pub fn pending_bytes(&self) -> usize

Get the number of pending bytes in the buffer.