Struct KvCache 

pub struct KvCache {
    pub n_kv_heads: usize,
    pub head_dim: usize,
    pub k: Vec<f32>,
    pub v: Vec<f32>,
    /* private fields */
}

Fields

n_kv_heads: usize

head_dim: usize

k: Vec<f32>

v: Vec<f32>

Implementations

impl KvCache

pub fn positions(&self) -> usize

Positions this sequence has consumed: what RoPE means, what a resume point means, and what a truncate target is measured in.

Monotonic except through Self::truncate and Self::clear. Equal to Self::rows today, and deliberately a different method so it stops being equal safely (#61).

pub fn rows(&self) -> usize

Rows of K/V actually resident: what attention iterates over and what the memory costs.

Derived from the buffer rather than counted alongside it, so it cannot drift from what is really there. That is the same rule the batched prefill learned in #37: read the cursor, do not keep a copy of it.

pub fn new(n_kv_heads: usize, head_dim: usize) -> Self

pub fn with_capacity( n_kv_heads: usize, head_dim: usize, max_seq_len: usize, ) -> Self

Pre-allocates storage for up to max_seq_len positions, so push never triggers a reallocation-and-copy during decode. Use this when the maximum context length is known ahead of time

pub fn with_pool( n_kv_heads: usize, head_dim: usize, pool: Arc<Mutex<KvBlockPool>>, max_seq_len: usize, ) -> Result<Self, KvPoolExhausted>

Acquires up front however many blocks from pool are needed to cover max_seq_len positions (at least one, even if max_seq_len is 0), so a caller that knows its worst-case sequence length ahead of time (as ferrox-server does: prompt length + max_tokens) never needs to acquire another block mid-decode. This matters beyond performance: push growing past its currently held capacity can fail if the pool is exhausted by other requests by then, and callers like ferrox_models::Decoder::forward_token treat push as infallible for non-pooled caches – a pooled cache that under-reserves at construction and then fails to grow later would violate that assumption and panic mid-decode. Sizing to max_seq_len up front turns that into an admission-control decision made once, honestly, before any generation work starts, exactly mirroring with_capacity’s worst-case pre-allocation – just drawn from a shared pool instead of a private allocation. Returns Err(KvPoolExhausted) without mutating anything if the pool doesn’t have that many blocks free.

pub fn push( &mut self, k_step: &[f32], v_step: &[f32], ) -> Result<(), KvPoolExhausted>

Appends one position’s key/value vectors (each n_kv_heads * head_dim long) to the cache. For pool-backed caches, this may need to acquire another block first; if the shared pool has none free, no data is appended and Err(KvPoolExhausted) is returned. Caches built with new or with_capacity always return Ok.

pub fn advance_len(&mut self, n: usize) -> Result<(), KvPoolExhausted>

Advance length by n positions without storing real K/V values (zero-fill). Used when Metal owns the KV plane and the host cache only needs matching seq_len for sync checks.

pub fn release_to_pool(&mut self)

Returns this cache’s blocks to its shared pool immediately (rather than waiting for Drop) and detaches it from pool accounting; a no-op for caches that aren’t pool-backed, and idempotent if called more than once.

pub fn clear(&mut self)

pub fn truncate(&mut self, new_seq_len: usize)

Rolls the cache back to exactly new_seq_len positions, discarding everything after. Used to reject speculatively decoded draft tokens that turned out wrong: their K/V were already pushed during batched verification, and rejection means removing them so the next real decode step continues from the last accepted position, not the last attempted one. Rolls the cache back to exactly new_seq_len POSITIONS.

A windowed cache can only roll back into rows it still holds. A target further back than Self::rows names a position this cache dropped behind its window, and there is no honest thing to return for it – so it stops, rather than silently rolling back to the oldest row it happens to have and answering the next token out of a history with a hole in it.

Unreachable for a cache that has not been armed by Self::arm_window, which is every cache unless FERROX_KV_WINDOW is on: rows == positions there, so the second precondition is implied by the first.

pub fn arm_window(&mut self, window: KvWindow)

Tells this cache it may drop rows that have fallen behind window (#61 step 2).

Arming alone drops nothing: Self::evict_behind_window is what drops, and the holder calls it at a point where it knows nothing is mid-read. That split is deliberate. push is the obvious place to evict and it is the wrong one: Decoder::forward_batch writes a whole prefill batch into the cache and only then reads it back, against a row offset it captured BEFORE the writes. Evicting inside push would move every row out from under that offset, and the prompt would be attended over shifted keys. So eviction is something the holder asks for, and asking for it too rarely only costs memory – over-retention is always correct, under-retention is wrong logits.

Idempotent; a second call with a different window replaces the first, and rows already dropped stay dropped.

pub fn window(&self) -> Option<KvWindow>

The window this cache evicts behind, or None if it keeps everything.

pub fn evict_behind_window(&mut self) -> usize

Drops rows that have fallen behind the armed window, returning how many rows went. Zero, always, for an unarmed cache.

The rows dropped are the OLDEST ones, so what remains is still a contiguous suffix of the sequence: row i of rows holds absolute position positions - rows + i. Every windowed attention kernel reads the last window rows and nothing else, and KvWindow::rows_after guarantees at least that many survive, so the set of rows a kernel reads is byte-for-byte the set it would have read with no eviction at all.

pub fn allocated_bytes(&self) -> usize

Bytes currently resident for this cache’s K and V buffers combined (actual allocated capacity, not just used length) – the number that matters for “does this fit in the context budget,”

pub fn is_within_planned_capacity(&self) -> bool

True if this cache was pre-allocated via with_capacity and has not yet grown past that planned capacity (i.e. push has never had to reallocate). Useful for tests/diagnostics confirming the pre-allocation path actually avoided reallocs.

Trait Implementations

impl Clone for KvCache

Cloning a pool-backed cache detaches the clone from pool accounting (its k/v/seq_len data is copied normally, but the clone does not hold or later release any blocks itself) – mirroring how ferrox-models::prefix_cache already uses KvCache::clone to fork a cached prefix into a new, independent request’s cache. Only the original cache’s blocks are released, exactly once, when it drops.

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Drop for KvCache

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.