Struct QuantValueCache 

pub struct QuantValueCache {
    pub d: usize,
    /* private fields */
}

Quantized Value cache: stores each appended token as D u8 bytes + 2 f32 params. Memory per token: D + 8 bytes (vs D*4 f32 = 4× saving at large D).

Fields

d: usize

Implementations

impl QuantValueCache

pub fn new(d: usize) -> Self

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn push_token(&mut self, v: &[f32])

Append one token’s V vector (length D), quantizing to u8.

pub fn dequant_all(&self) -> Array2<f32>

Dequantize all stored tokens to a [T, D] f32 array.

pub fn memory_bytes(&self) -> usize

Trait Implementations

impl Clone for QuantValueCache

fn clone(&self) -> QuantValueCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantValueCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for QuantValueCache

fn default() -> QuantValueCache

Returns the “default value” for a type.