Struct QuantKeyCache 

pub struct QuantKeyCache {
    pub d: usize,
    /* private fields */
}

Quantized Key cache: stores each appended token per-CHANNEL (each of the D channels has its own running scale accumulated across all tokens so far). On each new token, the channel scale may expand; old tokens in that channel are NOT re-quantized (acceptable error for a growing cache; exact re-quant is the follow-on). Memory: TD bytes + D2 f32 params.

Fields

d: usize

Implementations

impl QuantKeyCache

pub fn new(d: usize) -> Self

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn push_token(&mut self, k: &[f32])

Append one token’s K vector (length D), updating per-channel scales.

pub fn dequant_all(&self) -> Array2<f32>

Dequantize all stored tokens to a [T, D] f32 array.

pub fn memory_bytes(&self) -> usize

Trait Implementations

impl Clone for QuantKeyCache

fn clone(&self) -> QuantKeyCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantKeyCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for QuantKeyCache

fn default() -> QuantKeyCache

Returns the “default value” for a type.