Struct DecodeStream 

pub struct DecodeStream { /* private fields */ }

DecodeStream will keep the state necessary to produce individual chunks of strings given an input stream of token_ids.

This is necessary because decoding in general cannot achieve that since strings depend on surrounding ids to provide a valid string. Typically stripping extra spaces.

Implementations

impl DecodeStream

pub fn new( tokenizer: Arc<dyn Tokenizer>, prompt_token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Self

pub fn step(&mut self, id: u32) -> Result<Option<String>>

Step appends a token_id to the internal state and tries to produce a text chunk.

Implementation directly copied from Huggingface’s TGI: https://github.com/huggingface/text-generation-inference/blob/24c2bff65924801ddf90fa24fcc72752d4f45538/server/text_generation_server/models/model.py#L144

Returning None means the given id is not enough to produce a chunk. This typically happens with byte_fallback options where some tokens do not represent valid UTF-8, and only follow-up token_ids will help produce a valid chunk.