Struct QuantizedKvCache 

pub struct QuantizedKvCache {
    pub layers: Vec<QuantizedKvLayer>,
}

All layers of a quantized KV cache.

Fields

layers: Vec<QuantizedKvLayer>

Implementations

impl QuantizedKvCache

pub fn new(n_layers: usize, kv_dim: usize, scheme: KvQuant) -> Result<Self>

pub fn n_layers(&self) -> usize

pub fn past_len(&self) -> usize

pub fn bytes(&self) -> usize

Total bytes across all layers.

Trait Implementations

impl Clone for QuantizedKvCache

fn clone(&self) -> QuantizedKvCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantizedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.