Struct DecodeInput 

pub struct DecodeInput {
    pub input_ids: TensorRef,
    pub kv_cache: Arc<dyn KvCacheHandle>,
    pub position_ids: Option<TensorRef>,
}

Input for decode phase (generating one token at a time)

Fields

input_ids: TensorRef

Input token ID for current step [batch_size, 1]

kv_cache: Arc<dyn KvCacheHandle>

Existing KV cache from previous steps

position_ids: Option<TensorRef>

Position IDs for current step [batch_size, 1] (optional)

Implementations

impl DecodeInput

pub fn new(input_ids: TensorRef, kv_cache: Arc<dyn KvCacheHandle>) -> Self

Create new decode input

pub fn with_position_ids(self, positions: TensorRef) -> Self

Add position IDs

pub fn batch_size(&self) -> usize

Get batch size

Trait Implementations

impl Clone for DecodeInput

fn clone(&self) -> DecodeInput

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DecodeInput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.