Skip to main content

Decoder

dynamo_tokenizers::traits

Trait Decoder 

Source 
pub trait Decoder: Send + Sync {
    // Required method
    fn decode(
        &self,
        token_ids: &[TokenIdType],
        skip_special_tokens: bool,
    ) -> Result<DecodeResult>;
}
Expand description

Implementations must ensure that partial multi-byte sequences produce U+FFFD (\u{FFFD}) in the output rather than returning Err. This is commonly achieved via String::from_utf8_lossy (tiktoken) or library-internal byte-fallback handling (HuggingFace). DecodeStream::step() relies on DecodeResult::Partial to detect incomplete sequences and buffer tokens until the full character arrives.

Required Methods§

Source

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§