Struct Fp8KvCache 

pub struct Fp8KvCache {
    pub layers: Vec<Fp8KvLayer>,
}

Full multi-layer FP8-quantized KV cache for autoregressive decoding.

Wraps one Fp8KvLayer per transformer layer and exposes per-layer mutable and immutable accessors. All layers share the same format, num_kv_heads, head_dim, and capacity.

Fields

layers: Vec<Fp8KvLayer>

Per-transformer-layer FP8 KV stores.

Implementations

impl Fp8KvCache

pub fn new( num_layers: usize, num_kv_heads: usize, head_dim: usize, capacity: usize, format: Fp8KvFormat, ) -> Self

Allocate a new FP8 KV cache for num_layers transformer layers.

Each layer is pre-allocated for capacity token positions.

pub fn layer(&self, layer_idx: usize) -> &Fp8KvLayer

Immutable reference to a specific layer.

Panics

Panics if layer_idx >= self.num_layers().

pub fn layer_mut(&mut self, layer_idx: usize) -> &mut Fp8KvLayer

Mutable reference to a specific layer.

Panics

Panics if layer_idx >= self.num_layers().

pub fn num_layers(&self) -> usize

Number of transformer layers in this cache.

pub fn total_memory_bytes(&self) -> usize

Total memory used across all layers in bytes.

pub fn clear_all(&mut self)

Clear all layers, resetting stored lengths to zero.

Trait Implementations

impl Debug for Fp8KvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.