Struct GptqLinear 

pub struct GptqLinear<B: Backend> { /* private fields */ }

Implementations

impl<B: Backend> GptqLinear<B>

pub fn from_raw( qweight: &[i32], scales: &[f32], qzeros: &[i32], g_idx: Option<&[i32]>, bits: u32, group_size: usize, in_features: usize, out_features: usize, ) -> Result<Self>

Build from raw host-side GPTQ tensors. The Backend repacks into its preferred format once; inference uses the repacked store.

qweight: [k/8, n] i32 (packed int4) scales: [k/group_size, n] f32 (converted from f16 by caller) qzeros: [k/group_size, n/8] i32 g_idx: [k] i32 — optional, only used for desc_act=true

pub fn from_store( store: B::GptqStore, in_features: usize, out_features: usize, ) -> Self

Construct directly from a pre-built backend store (e.g. tests).

pub fn with_bias(self, bias: &[f32]) -> Self

Attach a bias vector ([out_features] f32 on host, uploaded to backend). Qwen2.5 / Llama-with-bias variants require this.

pub fn store(&self) -> &B::GptqStore

Trait Implementations

impl<B: Backend> Linear<B> for GptqLinear<B>

fn in_features(&self) -> usize

fn out_features(&self) -> usize

fn forward( &self, ctx: &mut B::Context, input: &B::Buffer, out: &mut B::Buffer, m: usize, )

Append GEMM work onto ctx. Caller flushes the context when results must be materialised.