Struct GptqLinear 

pub struct GptqLinear<B: Backend + BackendQuantMarlin> { /* private fields */ }

GPTQ-format Linear projection, polymorphic over backend.

Holds a boxed backend-specific Linear<B> produced by B::load_gptq. forward() delegates straight through.

Implementations

impl<B: Backend + BackendQuantMarlin> GptqLinear<B>

pub fn from_raw( qweight: &[i32], scales: &[f32], qzeros: &[i32], g_idx: Option<&[i32]>, bias: Option<&[f32]>, bits: u32, group_size: usize, in_features: usize, out_features: usize, ) -> Result<Self>

Build from raw host-side GPTQ tensors. The Backend repacks into its preferred format once (Marlin tiles on CUDA, dequant on CPU) and returns a boxed Linear; inference uses the boxed forward.

qweight: [k/8, n] i32 (packed int4) scales: [k/group_size, n] f32 (converted from f16 by caller) qzeros: [k/group_size, n/8] i32 g_idx: [k] i32 — optional, only used for desc_act=true bias: [n] f32 — optional fused bias (Qwen2.5 attention)

Trait Implementations

impl<B: Backend + BackendQuantMarlin> Linear<B> for GptqLinear<B>

fn in_features(&self) -> usize

fn out_features(&self) -> usize

fn forward( &self, ctx: &mut B::Context, input: &B::Buffer, out: &mut B::Buffer, m: usize, )

Append GEMM work onto ctx. Caller flushes the context when results must be materialised.