Struct PagedKVCache 

pub struct PagedKVCache<B: Backend> { /* private fields */ }

Implementations

impl<B: Backend> PagedKVCache<B>

pub fn new(num_layers: usize, config: CacheConfig) -> Self

Creates an empty paged cache for num_layers layers (all global). Arena tensors are allocated lazily on first use of each layer.

pub fn new_with_windows( num_layers: usize, config: CacheConfig, windows: Vec<Option<usize>>, ) -> Self

Creates a paged cache with a per-layer sliding-window assignment (windows[i] = Some(w) ⇒ layer i stores at most w-1 past tokens). w >= 2 — a window of 1 would leave decode steps with no past context at all.

pub fn num_free_pages(&self) -> usize

Number of free pages in the arena.

pub fn page_stats_inner(&self) -> PageStats

Page-table snapshot (see KVCache::page_stats).

Trait Implementations

impl<B: Backend> KVCache<B> for PagedKVCache<B>

fn popn(&mut self, n: usize) -> usize

Rolls back the last n cached tokens, freeing trailing pages that become fully unused. K/V content of popped positions is left in the arena but is never read (writes always cover seq_len.. densely).

Sliding layers can only roll back while nothing has been evicted from their window: once eviction starts, the tokens a rollback would re-expose are gone, so the whole cache refuses (0) and the caller rebuilds from scratch — the same “prefix caching disables under sliding windows” rule HF applies. All-or-nothing: state is only mutated when the full rollback is possible.

fn attention_opts( &mut self, layer: usize, q: Tensor<B, 4>, k: Tensor<B, 4>, v: Tensor<B, 4>, pos: usize, scale: f64, window: Option<usize>, ) -> Tensor<B, 4>

KVCache::attention with an optional sliding-window span (Gemma local layers): when Some(w), query at absolute position p attends only keys in (p - w, p] — older keys stay cached but are masked out. None = full causal attention (Llama-family behavior).

fn seq_len(&self) -> usize

Total cached sequence length.

fn reset(&mut self)

Drops all cached state (session reset).

fn pages_used(&self) -> Option<usize>

Pages currently allocated to the sequence (paged cache only).

fn page_stats(&self) -> Option<PageStats>

Page-table snapshot for observability (paged cache only). Cheap: reads counters, never touches device memory.

fn attention( &mut self, layer: usize, q: Tensor<B, 4>, k: Tensor<B, 4>, v: Tensor<B, 4>, pos: usize, scale: f64, ) -> Tensor<B, 4>

Appends seq new positions of K/V for layer and computes attention of q against the full cached window (past + new).