Struct KvCache 

pub struct KvCache<B: Backend> {
    pub k: B::Buffer,
    pub v: B::Buffer,
    pub len: usize,
    pub capacity: usize,
    pub num_kv_heads: usize,
    pub head_dim: usize,
}

Per-layer KV cache. Each model owns its own Vec<KvCache<B>> per sequence.

Fields

k: B::Buffer

v: B::Buffer

len: usize

capacity: usize

num_kv_heads: usize

head_dim: usize