Module detokenizer 

Incremental, byte-exact detokenization.

Byte-fallback tokenizers split a single UTF-8 codepoint across multiple tokens (common for emoji, CJK, and accented text). Converting each token to a String in isolation therefore yields invalid UTF-8 mid-sequence, and the attribute-filtering conversions (LlamaModel::token_to_bytes) drop control/byte pieces entirely.

StreamDetokenizer solves this for token-by-token generation loops: it accumulates the raw piece bytes from LlamaModel::token_to_raw_bytes and only emits complete UTF-8, retaining any trailing partial sequence until a later token completes it.

use llama_cpp_4::model::{LlamaModel, Special};
use llama_cpp_4::token::detokenizer::StreamDetokenizer;
let mut detok = StreamDetokenizer::new(model, Special::Plaintext);
let mut text = String::new();
for &token in tokens {
    text.push_str(&detok.push(token).unwrap());
}
text.push_str(&detok.finish().unwrap());

Structs

StreamDetokenizer

A stateful, incremental detokenizer built on LlamaModel::token_to_raw_bytes.

Enums

DetokenizeError

An error produced while streaming detokenization.