Struct LayerKvCache 

pub struct LayerKvCache {
    pub config: KVCacheConfig,
    /* private fields */
}

Cheap to clone (an Arc around the shared, lock-guarded buffer) — matches the shared-state pattern used elsewhere in this crate (e.g. ClusterState).

Fields

config: KVCacheConfig

Implementations

impl LayerKvCache

pub fn new(config: KVCacheConfig) -> Self

Creates an empty cache. Does not allocate the backing buffer yet — initialize() performs the single upfront allocation, sized once for config.max_tokens_per_seq.

pub fn initialize(&self, seq_len: usize) -> Result<(), String>

Ensures the backing buffer exists (allocating it on the first call only) and marks the first seq_len tokens (clamped to config.max_tokens_per_seq) as valid.

Safe to call again on the same cache — e.g. to grow current_len as more tokens are generated — without losing previously written data: the buffer is allocated once and never reallocated by this method.

pub fn write_kv( &self, token_idx: usize, keys: &[f32], values: &[f32], ) -> Result<(), String>

Writes one token’s key/value vectors into the cache at token_idx. Copies directly into the pre-allocated buffer slice — no allocation.

pub fn write_kv_batch( &self, entries: &[(usize, &[f32], &[f32])], ) -> Result<(), String>

Writes multiple tokens’ key/value vectors under a single write-lock acquisition, validating every entry upfront (before taking the lock) so a bad entry anywhere in the batch fails the whole call without partially writing it. Same per-token semantics as calling write_kv in a loop — this only changes how many times the lock is acquired.

pub fn read_kv(&self, token_idx: usize) -> Result<KVCacheEntry, String>

Reads one token’s cached key/value vectors as an owned, fixed-size copy (config.token_width() floats each — not the whole cache).

pub fn read_range( &self, start_token: usize, end_token: usize, ) -> Result<(Vec<f32>, Vec<f32>), String>

Reads keys/values for a contiguous token range ([start_token, end_token)) in one call — the shape attention actually wants (all prior positions at once), instead of end_token - start_token separate lock acquisitions and small allocations.

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

Trait Implementations

impl Clone for LayerKvCache

fn clone(&self) -> LayerKvCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LayerKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LayerKvCache

fn default() -> Self

Returns the “default value” for a type.