Struct KVCache 

pub struct KVCache { /* private fields */ }

KV cache for a single model.

Pre-allocates buffers for the maximum sequence length to avoid allocations during generation.

Implementations

impl KVCache

pub fn new(num_layers: usize, num_kv_heads: usize, head_dim: usize) -> Self

Create a new empty KV cache.

pub fn with_capacity( num_layers: usize, num_kv_heads: usize, head_dim: usize, max_seq_len: usize, ) -> Self

Create a new KV cache with pre-allocated capacity.

pub fn append(&mut self, layer: usize, k_data: &[f32], v_data: &[f32])

Append K and V vectors for the current token to a specific layer.

k_data and v_data should each have length num_kv_heads * head_dim.

pub fn advance(&mut self)

Advance the sequence position by one token. Call this after appending K/V data to all layers.

pub fn k(&self, layer: usize) -> &[f32]

Get the full K cache for a layer: [len * num_kv_heads * head_dim].

pub fn v(&self, layer: usize) -> &[f32]

Get the full V cache for a layer: [len * num_kv_heads * head_dim].

pub fn len(&self) -> usize

Current sequence length.

pub fn is_empty(&self) -> bool

Whether the cache is empty.

pub fn num_layers(&self) -> usize

Number of layers.

pub fn clear(&mut self)

Clear the cache for a new generation.

pub fn entry_size(&self) -> usize

Entry size (num_kv_heads * head_dim) per token per layer.

Trait Implementations

impl Clone for KVCache

fn clone(&self) -> KVCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KVCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.