Struct QuantizedKvLayer 

pub struct QuantizedKvLayer {
    pub num_kv_heads: usize,
    pub head_dim: usize,
    pub capacity: usize,
    pub len: usize,
    /* private fields */
}

A single layer’s INT8-quantized KV cache.

Memory layout for the INT8 data arrays uses the token-major order [token_pos * num_kv_heads * head_dim], so sequential decode steps append contiguous blocks. Scale arrays use [token_pos * num_kv_heads].

Fields

num_kv_heads: usize

Number of KV attention heads.

head_dim: usize

Dimension of each attention head.

capacity: usize

Maximum number of token positions pre-allocated.

len: usize

Number of token positions actually stored so far.

Implementations

impl QuantizedKvLayer

pub fn new(capacity: usize, num_kv_heads: usize, head_dim: usize) -> Self

Allocate an empty quantized KV layer with the given dimensions.

Pre-allocates all storage so that subsequent push calls do not allocate.

pub fn push(&mut self, keys: &[f32], values: &[f32]) -> Result<(), QuantKvError>

Append keys and values for the next token position.

keys must be a flat slice of shape [num_kv_heads * head_dim] (heads first, then dims). values must have the same shape.

Each head’s row is quantized independently with its own scale.

Errors

• QuantKvError::CapacityExceeded if self.len == self.capacity.
• QuantKvError::ShapeMismatch if keys or values length is wrong.

pub fn get_key( &self, token_pos: usize, head: usize, ) -> Result<Vec<f32>, QuantKvError>

Get dequantized keys for a specific token position and head.

Returns a Vec<f32> of length head_dim.

Errors

• QuantKvError::PositionOutOfRange if token_pos >= self.len.
• QuantKvError::HeadOutOfRange if head >= self.num_kv_heads.

pub fn get_value( &self, token_pos: usize, head: usize, ) -> Result<Vec<f32>, QuantKvError>

Get dequantized values for a specific token position and head.

Returns a Vec<f32> of length head_dim.

Errors

• QuantKvError::PositionOutOfRange if token_pos >= self.len.
• QuantKvError::HeadOutOfRange if head >= self.num_kv_heads.

pub fn get_keys_at(&self, token_pos: usize) -> Result<Vec<f32>, QuantKvError>

Get all dequantized keys for a token position (all heads, interleaved).

Returns a flat Vec<f32> of length num_kv_heads * head_dim.

Errors

• QuantKvError::PositionOutOfRange if token_pos >= self.len.

pub fn get_values_at(&self, token_pos: usize) -> Result<Vec<f32>, QuantKvError>

Get all dequantized values for a token position (all heads, interleaved).

Returns a flat Vec<f32> of length num_kv_heads * head_dim.

Errors

• QuantKvError::PositionOutOfRange if token_pos >= self.len.

pub fn memory_bytes(&self) -> usize

Memory used by this layer in bytes (INT8 data + f32 scales).

Only accounts for the pre-allocated storage slabs, not struct overhead.

pub fn fp32_memory_bytes(&self) -> usize

Equivalent memory if the same data were stored as FP32 (no scales).

2 * capacity * num_kv_heads * head_dim * 4 bytes

pub fn compression_ratio(&self) -> f32

Compression ratio versus FP32 storage.

Values approaching 4.0 indicate near-ideal INT8 compression. The ratio is slightly below 4.0 because per-row f32 scales add overhead.

Trait Implementations

impl Debug for QuantizedKvLayer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.