Struct LayerKvCache 

pub struct LayerKvCache {
    pub config: KVCacheConfig,
    /* private fields */
}

Cheap to clone (an Arc around the shared, lock-guarded buffer) — matches the shared-state pattern used elsewhere in this crate (e.g. ClusterState).

Fields

config: KVCacheConfig

Implementations

impl LayerKvCache

pub fn new(config: KVCacheConfig) -> Self

Creates an empty cache. Does not allocate the backing buffer yet — initialize() performs the single upfront allocation, sized once for config.max_tokens_per_seq.

pub fn initialize(&self, seq_len: usize) -> Result<(), String>

Ensures the backing buffer exists (allocating it on the first call only) and marks the first seq_len tokens (clamped to config.max_tokens_per_seq) as valid.

Safe to call again on the same cache — e.g. to grow current_len as more tokens are generated — without losing previously written data: the buffer is allocated once and never reallocated by this method.

pub fn write_kv( &self, token_idx: usize, keys: &[f32], values: &[f32], ) -> Result<(), String>

Writes one token’s key/value vectors into the cache at token_idx. Copies directly into the pre-allocated buffer slice — no allocation.

pub fn write_kv_batch( &self, entries: &[(usize, &[f32], &[f32])], ) -> Result<(), String>

Writes multiple tokens’ key/value vectors under a single write-lock acquisition, validating every entry upfront (before taking the lock) so a bad entry anywhere in the batch fails the whole call without partially writing it. Same per-token semantics as calling write_kv in a loop — this only changes how many times the lock is acquired.

pub fn with_read_kv<F, R>(&self, token_idx: usize, f: F) -> Result<R, String>

where F: FnOnce(&[f32], &[f32]) -> R,

Inspects one token’s cached key/value slices directly under a read lock via a closure. Zero heap allocation on this path.

pub fn with_read_range<F, R>( &self, start_token: usize, end_token: usize, f: F, ) -> Result<R, String>

where F: FnOnce(KVSpan<'_>) -> R,

Inspects keys/values for a contiguous token range ([start_token, end_token)) directly under a read lock via a closure that accepts a KVSpan. Zero heap allocation on this path.

pub fn read_range_into( &self, start_token: usize, end_token: usize, keys_out: &mut [f32], values_out: &mut [f32], ) -> Result<(), String>

Copies keys and values for a contiguous token range into pre-allocated caller slices keys_out and values_out. Rejects output buffers if they are too small for the requested token range.

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

Trait Implementations

impl Clone for LayerKvCache

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LayerKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LayerKvCache

fn default() -> Self

Returns the “default value” for a type.