pub struct KvCache {
pub seq_len: usize,
/* private fields */
}Expand description
Host-side KV cache with one K and one V buffer per transformer layer.
Layout per layer: contiguous f32 slice of length
context_len × num_kv_heads × head_dim.
Valid data occupies [0, seq_len × row_size).
Fields§
§seq_len: usizeImplementations§
Source§impl KvCache
impl KvCache
Sourcepub fn new(
num_layers: u32,
context_len: u32,
num_kv_heads: u32,
head_dim: u32,
) -> Self
pub fn new( num_layers: u32, context_len: u32, num_kv_heads: u32, head_dim: u32, ) -> Self
Allocate a zeroed cache for num_layers layers.
Sourcepub fn append(&mut self, layer: usize, k_row: &[f32], v_row: &[f32])
pub fn append(&mut self, layer: usize, k_row: &[f32], v_row: &[f32])
Append one token’s K and V vectors for a given layer.
When the cache is full, the oldest token is evicted (ring-buffer shift) before the new vectors are written.
Sourcepub fn get_k(&self, layer: usize) -> &[f32]
pub fn get_k(&self, layer: usize) -> &[f32]
Return the valid slice of K vectors for layer as [seq_len, row_size] data.
Sourcepub fn get_v(&self, layer: usize) -> &[f32]
pub fn get_v(&self, layer: usize) -> &[f32]
Return the valid slice of V vectors for layer as [seq_len, row_size] data.
Sourcepub fn get_k_at(&self, layer: usize, count: usize) -> &[f32]
pub fn get_k_at(&self, layer: usize, count: usize) -> &[f32]
Return the K slice for layer at an explicit token count.
Use this after append to include the just-written token whose position
has been filled but seq_len has not yet been incremented (because
seq_len only advances after the final layer has appended).
Sourcepub fn get_v_at(&self, layer: usize, count: usize) -> &[f32]
pub fn get_v_at(&self, layer: usize, count: usize) -> &[f32]
Return the V slice for layer at an explicit token count.
Sourcepub fn num_layers(&self) -> usize
pub fn num_layers(&self) -> usize
Total number of layers.
Sourcepub fn context_len(&self) -> usize
pub fn context_len(&self) -> usize
Maximum sequence length before eviction occurs.