Struct QuantizedKvCache 

pub struct QuantizedKvCache {
    pub num_layers: usize,
    pub num_kv_heads: usize,
    pub head_dim: usize,
    /* private fields */
}

Full multi-layer INT8-quantized KV cache for autoregressive decoding.

Wraps one QuantizedKvLayer per transformer layer and exposes a unified decode-step interface through push_step.

Fields

num_layers: usize

Number of transformer layers.

num_kv_heads: usize

Number of KV attention heads per layer.

head_dim: usize

Dimension of each attention head.

Implementations

impl QuantizedKvCache

pub fn new( num_layers: usize, capacity: usize, num_kv_heads: usize, head_dim: usize, ) -> Self

Allocate a new quantized KV cache for num_layers transformer layers.

Each layer is pre-allocated for capacity token positions.

pub fn push_step( &mut self, all_keys: &[Vec<f32>], all_values: &[Vec<f32>], ) -> Result<(), QuantKvError>

Append KV tensors for all layers at the current decode step.

all_keys[layer] must be a flat slice of shape [num_kv_heads * head_dim]. all_values[layer] must have the same shape.

Errors

• QuantKvError::LayerOutOfRange if all_keys.len() != self.num_layers.
• Propagates QuantKvError from each layer’s push.

pub fn get_key( &self, layer: usize, token_pos: usize, head: usize, ) -> Result<Vec<f32>, QuantKvError>

Get dequantized keys for a specific layer, token position, and head.

Errors

• QuantKvError::LayerOutOfRange if layer >= self.num_layers.
• Propagates position/head errors from the underlying layer.

pub fn get_value( &self, layer: usize, token_pos: usize, head: usize, ) -> Result<Vec<f32>, QuantKvError>

Get dequantized values for a specific layer, token position, and head.

Errors

• QuantKvError::LayerOutOfRange if layer >= self.num_layers.
• Propagates position/head errors from the underlying layer.

pub fn total_memory_bytes(&self) -> usize

Total memory used across all layers in bytes.

pub fn total_fp32_memory_bytes(&self) -> usize

FP32-equivalent memory across all layers.

pub fn compression_ratio(&self) -> f32

Overall compression ratio vs FP32.

pub fn seq_len(&self) -> usize

Number of token positions currently stored (taken from layer 0).

Returns 0 if there are no layers.

Trait Implementations

impl Debug for QuantizedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.