Struct KvCache 

pub struct KvCache {
    pub keys: Vec<ArrayD<f64>>,
    pub values: Vec<ArrayD<f64>>,
    pub seq_len: usize,
    pub max_seq_len: usize,
    pub num_layers: usize,
    pub num_heads: usize,
}

Cached key-value pairs for autoregressive inference using ndarray tensors.

Stores per-layer key and value tensors as dynamic-rank ArrayD<f64>. Tensors are concatenated along the sequence dimension on each append_kv call.

Fields

keys: Vec<ArrayD<f64>>

Cached K tensors per layer (shape: [seq_len, num_heads, head_dim] after appends).

values: Vec<ArrayD<f64>>

Cached V tensors per layer (shape: [seq_len, num_heads, head_dim] after appends).

seq_len: usize

Current cached sequence length.

max_seq_len: usize

Maximum allowed sequence length.

num_layers: usize

Number of transformer layers.

num_heads: usize

Number of attention heads.

Implementations

impl KvCache

pub fn new( num_layers: usize, num_heads: usize, head_dim: usize, max_seq_len: usize, ) -> Self

Create a new, empty KV-cache.

Initially all per-layer tensors are zero-sized along the sequence dimension.

pub fn append_kv( &mut self, layer: usize, new_k: ArrayD<f64>, new_v: ArrayD<f64>, ) -> Result<(), KvCacheError>

Append new key and value tensors for the given layer.

new_k and new_v must have shape [new_tokens, num_heads, head_dim].

pub fn get_kv(&self, layer: usize) -> Option<(&ArrayD<f64>, &ArrayD<f64>)>

Retrieve cached keys and values for the given layer.

Returns None if the layer index is out of bounds.

pub fn reset(&mut self)

Reset the cache to empty (seq_len == 0).

pub fn current_len(&self) -> usize

Current cached sequence length.

pub fn is_full(&self) -> bool

Returns true if the cache is at maximum capacity.

pub fn memory_usage_bytes(&self) -> usize

Approximate memory usage in bytes (f64 = 8 bytes per element).

Trait Implementations

impl Clone for KvCache

fn clone(&self) -> KvCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.