Struct PagedKvCache 

pub struct PagedKvCache { /* private fields */ }

A vLLM-style block-paged KV-cache. Instead of one contiguous pre-allocated tensor (which may fragment or require realloc), this manages memory in fixed-size 16-token blocks via a BlockTable.

Benefits:

• No single large allocation — blocks are page-sized
• Zero reallocation on append (new blocks allocated on demand from pool)
• Logical-to-physical mapping via block table
• Each block is independently cache-line friendly

Implementations

impl PagedKvCache

pub fn new(max_tokens: usize, dim: usize) -> Self

Create a paged KV-cache for max_tokens tokens of dimension dim.

Pre-allocates all blocks upfront to avoid any heap allocation during the inference loop. The number of blocks = ceil(max_tokens / 16).

pub fn len(&self) -> usize

Number of tokens currently stored.

pub fn is_empty(&self) -> bool

Whether no tokens are stored.

pub fn max_tokens(&self) -> usize

Maximum tokens this cache can hold.

pub fn dim(&self) -> usize

Hidden dimension per token.

pub fn num_blocks(&self) -> usize

Number of blocks allocated.

pub fn blocks_in_use(&self) -> usize

Number of blocks currently in use (partially or fully).

pub fn append(&mut self, token: &[f64]) -> Result<(), RuntimeError>

Append a single token vector. Zero allocation — writes into the next available slot in the current block.

pub fn append_tensor(&mut self, t: &Tensor) -> Result<(), RuntimeError>

Append a batch of tokens from a 2D tensor [n, dim].

pub fn as_tensor(&self) -> Tensor

Materialize all stored tokens into a contiguous Tensor [current_len, dim].

This is a read operation that copies data from blocks into a flat buffer. The copy is required since blocks are non-contiguous.

pub fn clear(&mut self)

Reset the cache to empty without deallocating blocks. Block data is retained; only cursors are reset.

pub fn get_token(&self, idx: usize) -> Result<Vec<f64>, RuntimeError>

Read a single token at logical position idx.

Trait Implementations

impl Clone for PagedKvCache

fn clone(&self) -> PagedKvCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PagedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for PagedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.