Trait Decoder 

pub trait Decoder: Send + Sync {
    // Required method
    fn decode(
        &self,
        token_ids: &[TokenIdType],
        skip_special_tokens: bool,
    ) -> Result<String>;

    // Provided method
    fn decode_step(
        &self,
        token_id: TokenIdType,
        ids: &mut Vec<TokenIdType>,
        prefix: &mut String,
        prefix_index: &mut usize,
        skip_special_tokens: bool,
    ) -> Result<Option<String>> { ... }
}

Core decoding trait - can be implemented independently

Required Methods

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<String>

Provided Methods

fn decode_step( &self, token_id: TokenIdType, ids: &mut Vec<TokenIdType>, prefix: &mut String, prefix_index: &mut usize, skip_special_tokens: bool, ) -> Result<Option<String>>

Incremental decode step — called once per generated token.

Maintains mutable state (ids, prefix, prefix_index) across calls to produce incremental text output. The default implementation uses the double-decode algorithm (decode prefix, decode prefix+new, diff).

HuggingFace overrides this with the native step_decode_stream from the tokenizers crate, which uses the same algorithm internally but avoids trait-method overhead for the two decode() calls.

Implementors

impl Decoder for CachedTokenizer

impl Decoder for HuggingFaceTokenizer

impl Decoder for MockTokenizer

impl Decoder for TiktokenTokenizer