Struct KvCache 

pub struct KvCache { /* private fields */ }

Per-layer, growable key/value cache for one generation session.

A fresh KvCache holds zero cached positions. Every call to crate::traits::Model::forward appends that call’s new positions to every layer’s cache (via KvCache::append, one call per layer) and reads back the full accumulated K/V for attention. The cache is specific to one generation session — call KvCache::new again (or KvCache::clear) to start a new, unrelated prompt.

Implementations

impl KvCache

pub fn new(n_layers: usize, max_context: usize) -> Self

A fresh, empty cache for a model with n_layers transformer blocks and a max_context-token context window (from crate::config::QwenConfig::max_context).

pub fn len(&self) -> usize

Number of positions currently cached (the same for every layer, since every KvCache::append call for a given forward pass appends the same number of new positions to each layer in turn). 0 for a fresh cache.

pub fn is_empty(&self) -> bool

pub fn max_context(&self) -> usize

pub fn clear(&mut self)

Drops every layer’s cached K/V, returning this cache to its freshly-KvCache::newd state so it can be reused for a new, unrelated prompt without reallocating the outer Vec.