Struct StreamDetokenizer 

pub struct StreamDetokenizer<'a> { /* private fields */ }

A stateful, incremental detokenizer built on LlamaModel::token_to_raw_bytes.

Feed tokens one at a time with push; each call returns the text that has become complete, buffering any partial multi-byte character until it can be finished. Call finish at the end of a generation to flush and validate any remaining bytes.

Implementations

impl<'a> StreamDetokenizer<'a>

pub fn new(model: &'a LlamaModel, special: Special) -> Self

Create a new streaming detokenizer over model.

special is forwarded to LlamaModel::token_to_raw_bytes and controls whether special/control tokens are rendered (Special::Tokenize) or treated as plaintext (Special::Plaintext).

pub fn push(&mut self, token: LlamaToken) -> Result<String, DetokenizeError>

Feed a single token, returning any text that is now complete.

A trailing incomplete UTF-8 sequence is retained internally and emitted once a subsequent token completes it.

Errors

• DetokenizeError::TokenToBytes if the token cannot be converted.
• DetokenizeError::InvalidUtf8 if the accumulated bytes contain a genuinely malformed (not merely incomplete) UTF-8 sequence.

pub fn push_all( &mut self, tokens: impl IntoIterator<Item = LlamaToken>, ) -> Result<String, DetokenizeError>

Feed several tokens, returning the concatenation of all completed text.

Errors

See push.

pub fn pending(&self) -> &[u8]

The bytes currently buffered awaiting completion of a multi-byte character. Empty when the stream is on a UTF-8 boundary.

pub fn finish(self) -> Result<String, DetokenizeError>

Finish the stream, returning any remaining complete text.

Errors

DetokenizeError::IncompleteUtf8 if bytes remain that do not form a complete UTF-8 sequence (carrying the leftover bytes), or DetokenizeError::InvalidUtf8 if the buffered bytes are malformed.

Trait Implementations

impl<'a> Debug for StreamDetokenizer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.