Struct Q40Weight 

pub struct Q40Weight<R: Runtime> { /* private fields */ }

A weight matrix resident in VRAM in packed Q4_0 form. [n_out, k] row-major, k % 32 == 0, blocks along k — exactly the GGUF tensor layout, so from_gguf_bytes takes the mmap’d tensor bytes unchanged.

Implementations

impl<R: Runtime> Q40Weight<R>

pub fn from_gguf_bytes( client: &ComputeClient<R>, data: &[u8], n_out: usize, k: usize, ) -> Result<Self>

Repacks a GGUF Q4_0 tensor onto the device. data is the raw block stream for an [n_out, k] weight (the bytes GgufSource maps).

pub fn n_out(&self) -> usize

Output features.

pub fn k(&self) -> usize

Input features.

pub fn vram_bytes(&self) -> usize

Bytes this weight occupies in VRAM (packed nibbles + f32 scales) — 20 bytes per 32 weights, vs 128 for f32 (6.4×) or 64 for f16 (3.2×).

pub fn matmul_device( &self, client: &ComputeClient<R>, x: Handle, m: usize, ) -> Handle

Device path: y = x @ W^T with x already resident as a contiguous f32 buffer of [m, k]. Launch only — returns the output handle ([m, n_out] f32) without any host round-trip.

pub fn matmul_host( &self, client: &ComputeClient<R>, x: &[f32], m: usize, ) -> Result<Vec<f32>>

y = x @ W^T for host-side x: [m, k], returning [m, n_out]. Host-slice convenience for tests/CLI probes.