Struct DecodeOutput 

pub struct DecodeOutput {
    pub logits: TensorRef,
    pub kv_cache: Arc<dyn KvCacheHandle>,
    pub recurrent_state: Option<Arc<dyn RecurrentStateHandle>>,
    pub hidden_state: Option<TensorRef>,
    pub attention_weights: Option<Vec<TensorRef>>,
}

Output from decode phase

Fields

logits: TensorRef

Logits for next token [batch_size, vocab_size]

kv_cache: Arc<dyn KvCacheHandle>

Updated KV cache with new token state

recurrent_state: Option<Arc<dyn RecurrentStateHandle>>

Updated recurrent state, when used.

hidden_state: Option<TensorRef>

Hidden state for current token (optional)

attention_weights: Option<Vec<TensorRef>>

Attention weights for current token (optional)

Implementations

impl DecodeOutput

pub fn new(logits: TensorRef, kv_cache: Arc<dyn KvCacheHandle>) -> Self

Create new decode output

pub fn with_recurrent_state( self, recurrent_state: Arc<dyn RecurrentStateHandle>, ) -> Self

Attach updated recurrent state to the decode output.

Trait Implementations

impl Clone for DecodeOutput

fn clone(&self) -> DecodeOutput

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DecodeOutput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.