Struct KvCache 

pub struct KvCache {
    pub n_kv_heads: usize,
    pub head_dim: usize,
    pub k: Vec<f32>,
    pub v: Vec<f32>,
    pub seq_len: usize,
    /* private fields */
}

Fields

n_kv_heads: usize

head_dim: usize

k: Vec<f32>

v: Vec<f32>

seq_len: usize

Implementations

impl KvCache

pub fn new(n_kv_heads: usize, head_dim: usize) -> Self

pub fn with_capacity( n_kv_heads: usize, head_dim: usize, max_seq_len: usize, ) -> Self

Pre-allocates storage for up to max_seq_len positions, so push never triggers a reallocation-and-copy during decode. Use this when the maximum context length is known ahead of time

pub fn with_pool( n_kv_heads: usize, head_dim: usize, pool: Arc<Mutex<KvBlockPool>>, max_seq_len: usize, ) -> Result<Self, KvPoolExhausted>

Acquires up front however many blocks from pool are needed to cover max_seq_len positions (at least one, even if max_seq_len is 0), so a caller that knows its worst-case sequence length ahead of time (as ferrox-server does: prompt length + max_tokens) never needs to acquire another block mid-decode. This matters beyond performance: push growing past its currently held capacity can fail if the pool is exhausted by other requests by then, and callers like ferrox_models::Decoder::forward_token treat push as infallible for non-pooled caches – a pooled cache that under-reserves at construction and then fails to grow later would violate that assumption and panic mid-decode. Sizing to max_seq_len up front turns that into an admission-control decision made once, honestly, before any generation work starts, exactly mirroring with_capacity’s worst-case pre-allocation – just drawn from a shared pool instead of a private allocation. Returns Err(KvPoolExhausted) without mutating anything if the pool doesn’t have that many blocks free.

pub fn push( &mut self, k_step: &[f32], v_step: &[f32], ) -> Result<(), KvPoolExhausted>

Appends one position’s key/value vectors (each n_kv_heads * head_dim long) to the cache. For pool-backed caches, this may need to acquire another block first; if the shared pool has none free, no data is appended and Err(KvPoolExhausted) is returned. Caches built with new or with_capacity always return Ok.

pub fn advance_len(&mut self, n: usize) -> Result<(), KvPoolExhausted>

Advance length by n positions without storing real K/V values (zero-fill). Used when Metal owns the KV plane and the host cache only needs matching seq_len for sync checks.

pub fn release_to_pool(&mut self)

Returns this cache’s blocks to its shared pool immediately (rather than waiting for Drop) and detaches it from pool accounting; a no-op for caches that aren’t pool-backed, and idempotent if called more than once.

pub fn clear(&mut self)

pub fn truncate(&mut self, new_seq_len: usize)

Rolls the cache back to exactly new_seq_len positions, discarding everything after. Used to reject speculatively decoded draft tokens that turned out wrong: their K/V were already pushed during batched verification, and rejection means removing them so the next real decode step continues from the last accepted position, not the last attempted one.

pub fn allocated_bytes(&self) -> usize

Bytes currently resident for this cache’s K and V buffers combined (actual allocated capacity, not just used length) – the number that matters for “does this fit in the context budget,”

pub fn is_within_planned_capacity(&self) -> bool

True if this cache was pre-allocated via with_capacity and has not yet grown past that planned capacity (i.e. push has never had to reallocate). Useful for tests/diagnostics confirming the pre-allocation path actually avoided reallocs.

Trait Implementations

impl Clone for KvCache

Cloning a pool-backed cache detaches the clone from pool accounting (its k/v/seq_len data is copied normally, but the clone does not hold or later release any blocks itself) – mirroring how ferrox-models::prefix_cache already uses KvCache::clone to fork a cached prefix into a new, independent request’s cache. Only the original cache’s blocks are released, exactly once, when it drops.

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Drop for KvCache

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.