Struct ContiguousKVCache 

pub struct ContiguousKVCache<B: Backend> { /* private fields */ }

Simple contiguous cache: stores one K and one V tensor per layer and concatenates along the sequence dimension every step.

Cost: an O(seq) copy per token per layer — kept as the correctness baseline; the paged arena is the production default.

Implementations

impl<B: Backend> ContiguousKVCache<B>

pub fn new(num_layers: usize) -> Self

Creates an empty cache for num_layers layers.

Trait Implementations

impl<B: Backend> KVCache<B> for ContiguousKVCache<B>

fn attention_opts( &mut self, layer: usize, q: Tensor<B, 4>, k: Tensor<B, 4>, v: Tensor<B, 4>, pos: usize, scale: f64, window: Option<usize>, ) -> Tensor<B, 4>

KVCache::attention with an optional sliding-window span (Gemma local layers): when Some(w), query at absolute position p attends only keys in (p - w, p] — older keys stay cached but are masked out. None = full causal attention (Llama-family behavior).

fn seq_len(&self) -> usize

Total cached sequence length.

fn reset(&mut self)

Drops all cached state (session reset).

fn attention( &mut self, layer: usize, q: Tensor<B, 4>, k: Tensor<B, 4>, v: Tensor<B, 4>, pos: usize, scale: f64, ) -> Tensor<B, 4>

Appends seq new positions of K/V for layer and computes attention of q against the full cached window (past + new).

fn popn(&mut self, n: usize) -> usize

Rolls back the last n cached tokens, returning how many were actually dropped. Caches that cannot roll back (the contiguous baseline) return 0 — callers gate prefix reuse on a nonzero result.

fn pages_used(&self) -> Option<usize>

Pages currently allocated to the sequence (paged cache only).

fn page_stats(&self) -> Option<PageStats>

Page-table snapshot for observability (paged cache only). Cheap: reads counters, never touches device memory.