Struct QuantizedWeight 

pub struct QuantizedWeight {
    pub data: Vec<u8>,
    pub scales: Vec<f32>,
    pub in_feat: usize,
    pub out_feat: usize,
    pub num_groups: usize,
}

Packed INT4 weight tensor with per-group f32 scales.

data: row-major packed nibbles [pair * out_feat + o], total (in_padded / 2) * out_feat bytes. scales: [num_groups * out_feat] f32, row-major [g * out_feat + o]. in_feat: original (unpadded) input dimension. out_feat: output dimension.

Fields

data: Vec<u8>

scales: Vec<f32>

in_feat: usize

out_feat: usize

num_groups: usize

Implementations

impl QuantizedWeight

pub fn quantize(w: &[f32], in_feat: usize, out_feat: usize) -> Self

Quantize an f32 weight matrix w of shape [in_feat, out_feat]. Matches Python _fake_quantize_int4 exactly.

pub fn dequantize_to(&self, out: &mut [f32])

Dequantize to f32 into out buffer of shape [in_feat, out_feat].

pub fn matvec(&self, x: &[f32], y: &mut [f32])

Matrix-vector multiply: y = W^T x (y shape [out_feat], x shape [in_feat]). Dequantizes on the fly — never materializes the full weight matrix. Uses AVX2 auto-vectorized inner loop on x86_64 when the feature is present.

pub fn matmul(&self, x: &[f32], batch: usize, y: &mut [f32])

Batched matmul: Y = X W (X shape [batch, in_feat], Y shape [batch, out_feat]).