Skip to main content

Decoder

dynamo_tokenizers::traits

Trait Decoder 

Source 
pub trait Decoder: Send + Sync {
    // Required method
    fn decode(
        &self,
        token_ids: &[TokenIdType],
        skip_special_tokens: bool,
    ) -> Result<DecodeResult>;

    // Provided method
    fn has_unstable_suffix(
        &self,
        _token_ids: &[TokenIdType],
        _skip_special_tokens: bool,
    ) -> bool { ... }
}
Expand description

Implementations must ensure that partial multi-byte sequences produce U+FFFD (\u{FFFD}) in the output rather than returning Err. This is commonly achieved via String::from_utf8_lossy (tiktoken) or library-internal byte-fallback handling (HuggingFace). DecodeStream::step() relies on DecodeResult::Partial to detect incomplete sequences and buffer tokens until the full character arrives.

Required Methods§

Source

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

Provided Methods§

Source

fn has_unstable_suffix( &self, _token_ids: &[TokenIdType], _skip_special_tokens: bool, ) -> bool

Whether appending tokens can still reinterpret the decoded suffix. Callers must buffer an unstable suffix until a boundary or end of input.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§