Struct Linear4Bit 

pub struct Linear4Bit {
    pub weight_packed: Tensor,
    pub scales: Tensor,
    pub bias: Option<Tensor>,
    pub group_size: usize,
    pub in_features: usize,
    pub out_features: usize,
}

4-bit quantized linear layer.

4ビット量子化された線形層。 グループごとのスケーリングにより精度を保ちつつメモリ使用量を削減します。

Stores weights in 4-bit format with group-wise scaling to maintain accuracy while reducing memory usage.

Fields

weight_packed: Tensor

Packed weights: [out_dim, in_dim/2] as U8 各バイトに2つの4ビット重みを格納

scales: Tensor

Per-group scales: [out_dim, n_groups] as F16 グループごとのスケール係数

bias: Option<Tensor>

Optional bias: [out_dim]

group_size: usize

Group size for quantization (typically 64 or 128)

in_features: usize

Input feature dimension

out_features: usize

Output feature dimension

Implementations

impl Linear4Bit

pub fn new( weight_packed: Tensor, scales: Tensor, bias: Option<Tensor>, group_size: usize, in_features: usize, out_features: usize, ) -> Result<Self>

Create a new Linear4Bit layer with pre-computed packed weights.

事前計算されたパック済み重みで新しいLinear4Bit層を作成。

Arguments / 引数

• weight_packed: Packed weights [out_dim, in_dim/2] as U8
• scales: Per-group scales [out_dim, n_groups] as F16
• bias: Optional bias [out_dim]
• group_size: Size of each quantization group
• in_features: Input dimension
• out_features: Output dimension

pub fn load_4bit(vb: &VarBuilder<'_>, prefix: &str) -> Result<Self>

Load 4-bit quantized weights from VarBuilder.

VarBuilderから4ビット量子化重みをロード。

Arguments / 引数

• vb: VarBuilder for loading weights
• prefix: Prefix for weight names (e.g., “layers.0.mlp.gate_proj”)

Expected files / 期待するファイル

• {prefix}.weight_4bit: Packed weights [out_dim, in_dim/2] as U8
• {prefix}.scales_4bit: Per-group scales [out_dim, n_groups] as F16
• {prefix}.bias (optional): Bias [out_dim]

pub fn load_direct( tensors: &HashMap<String, Tensor>, prefix: &str, in_dim: usize, out_dim: usize, _group_size: usize, _symmetric: bool, device: &Device, ) -> Result<Self>

Load from pre-loaded tensor HashMap (bypasses VarBuilder).

事前ロードしたテンソルHashMapから直接ロードします（VarBuilderをバイパス）。

Arguments / 引数

• tensors: Pre-loaded tensor HashMap
• prefix: Prefix for weight names (e.g., “layers.0.mlp.gate_proj”)
• in_dim: Input dimension
• out_dim: Output dimension
• group_size: Group size for quantization (use 128 as default)
• _symmetric: Whether quantization is symmetric (currently unused)
• device: Target device

pub fn forward(&self, input: &Tensor) -> Result<Tensor>

Forward pass: Y = X @ W^T + b

順伝播: Y = X @ W^T + b

Uses fused 4-bit GEMM for better performance. Automatically routes to CUDA kernel when input is on GPU.

Arguments / 引数

• input: Input tensor […, in_features]

Returns / 戻り値

Output tensor […, out_features]

pub fn forward_unpack(&self, input: &Tensor) -> Result<Tensor>

Forward pass with weight unpacking (legacy path).

This method unpacks weights to FP32 before matmul. Use forward() for better performance with fused GEMM.

Trait Implementations

impl Clone for Linear4Bit

fn clone(&self) -> Linear4Bit

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Linear4Bit

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.