Struct PagedKvCache 

pub struct PagedKvCache { /* private fields */ }

One sequence’s view into a shared PagedKvStore: a block table (which physical blocks this sequence’s positions live in, in order) plus how many positions have been written so far. Unlike KvCache, this holds no K/V data itself – every read and write goes through the shared store.

Implementations

impl PagedKvCache

pub fn new() -> Self

pub fn seq_len(&self) -> usize

pub fn block_table(&self) -> &[usize]

pub fn push( &mut self, store: &mut PagedKvStore, k_step: &[f32], v_step: &[f32], ) -> Result<(), PagedStoreExhausted>

Appends one position’s key/value vectors, acquiring a new block from store first if the current tail block is full (or none held yet). Mirrors KvCache::push’s signature/semantics exactly, just against shared storage instead of a private buffer.

pub fn release(&mut self, store: &mut PagedKvStore)

Releases every block this sequence holds back to store. Must be called explicitly (there’s no Drop here, since dropping needs a &mut PagedKvStore this type doesn’t own a reference to) – mirrors KvCache::release_to_pool, just not automatic.

pub fn blocks_needed_for(&self, store: &PagedKvStore, n_new: usize) -> usize

How many additional blocks appending n_new positions would take from store, given what this sequence already holds.

Counted against held CAPACITY rather than against seq_len, so it is right in both cases. The tail block is usually part-full, so the answer is never simply n_new / block_size: positions that land in a block already held cost nothing. And a sequence that pre-reserved (see Self::reserve) holds blocks beyond its length, which a seq_len-only sum would ask for twice.

Callers that must not fail part-way through a write check this against PagedKvStore::free_block_count before touching anything.

pub fn reserve( &mut self, store: &mut PagedKvStore, n_new: usize, ) -> Result<(), PagedStoreExhausted>

Takes the blocks n_new more positions will need, without advancing seq_len.

This is what makes a multi-layer append all-or-nothing. The check and the taking happen together, so every later Self::push writes into a block this sequence already owns and cannot fail. Reserving and then not filling is harmless: the blocks are this sequence’s until it releases, and seq_len still says how far it really got.

pub fn adopt_blocks( &mut self, block_table: Vec<usize>, seq_len: usize, block_size: usize, )

Installs a block table the caller allocated, with seq_len positions already computed in it.

This is how a sequence starts life on top of a cached prefix: the blocks are somebody else’s, already full, and this sequence appends past them. seq_len MUST be a whole number of blocks, because the first append writes at seq_len and a shared block must never be written – another sequence is attending over it. A ragged length would put that write inside the last shared block, corrupting a prefix every other holder is reading.

pub fn to_contiguous(&self, store: &PagedKvStore) -> KvCache

Copies this sequence’s KV out of the shared store into a plain contiguous KvCache.

This is what lets the batched prefill path run unchanged over paged storage. Its fast arm hands cache.k / cache.v to a blocked kernel that reads them as flat slices, and a block table cannot be expressed that way. Rather than maintain a second prefill kernel that reads through the table – a copy that could drift from the one every other model path uses – the pages are materialised once per layer, the existing kernel runs, and the new rows go back with Self::append_contiguous.

The cost is one seq_len * n_kv_heads * head_dim copy per layer per prefill call, against matmuls that dominate prefill. Decode still reads through the block table and copies nothing, which is where page sharing actually pays.

pub fn append_contiguous( &mut self, store: &mut PagedKvStore, k: &[f32], v: &[f32], count: usize, ) -> Result<(), PagedStoreExhausted>

Appends count positions’ worth of contiguous K/V rows, the inverse of Self::to_contiguous.

Blocks are reserved for the whole append before the first row is written, so a store that cannot hold the request refuses it having changed nothing. Writing rows until the store runs dry would leave the sequence with a seq_len that disagrees with the model’s own idea of how far it has got, which is not a recoverable state.

Trait Implementations

impl Clone for PagedKvCache

fn clone(&self) -> PagedKvCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PagedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PagedKvCache

fn default() -> PagedKvCache

Returns the “default value” for a type.