Struct QuantizedLinear 

pub struct QuantizedLinear { /* private fields */ }

A linear layer with quantized base weights and trainable LoRA adapters.

Dequantization Modes

• On-the-fly (default): Dequantizes during each forward pass, saves memory.
• Cached (opt-in via cache_dequantized): Dequantizes once, faster inference.

For training, always use on-the-fly mode (default) to save memory. For inference, consider enabling caching for ~30% speedup.

Implementations

impl QuantizedLinear

pub fn from_weight( weight: &Tensor, bias: Option<Tensor>, config: &QLoraConfig, device: &Device, ) -> Result<QuantizedLinear, QLoraError>

Create a new quantized linear layer from existing weights.

Uses on-the-fly dequantization by default (memory-optimal). Set config.cache_dequantized = true for inference speedup.

Arguments

• weight - Full-precision weight tensor to quantize
• bias - Optional bias tensor (kept in full precision)
• config - QLoRA configuration
• device - Device for computation

Errors

Returns error if weight tensor has invalid shape or quantization fails

pub fn from_weight_with_varbuilder( weight: &Tensor, bias: Option<Tensor>, config: &QLoraConfig, vb: VarBuilderArgs<'_, Box<dyn SimpleBackend + '_>>, ) -> Result<QuantizedLinear, QLoraError>

Create a quantized linear layer with trainable LoRA weights registered via VarBuilder.

This constructor ensures LoRA A/B weights are tracked for gradient computation. Use this for training; use from_weight for inference.

Arguments

• weight - Full-precision weight tensor to quantize
• bias - Optional bias tensor (kept in full precision)
• config - QLoRA configuration
• vb - VarBuilder backed by VarMap for gradient tracking

Errors

Returns error if weight tensor has invalid shape or quantization fails

pub fn new( in_features: usize, out_features: usize, config: &QLoraConfig, device: &Device, ) -> Result<QuantizedLinear, QLoraError>

Create a new quantized linear layer with zero-initialized quantized weights.

Primarily for testing; use from_weight for actual models.

Errors

Returns error if tensor creation or quantization fails

pub fn forward(&self, input: &Tensor) -> Result<Tensor, QLoraError>

Forward pass through the quantized linear layer.

Computes: output = x @ W_q^T + x @ (B @ A)^T * scaling + bias

Uses on-the-fly dequantization unless cache_dequantized was enabled.

Errors

Returns error if tensor operations fail

pub fn enable_weight_caching(&mut self) -> Result<(), QLoraError>

Enable weight caching for faster inference.

Call this after loading a trained model for inference. Not recommended for training (wastes memory).

Errors

Returns error if dequantization fails.

pub fn disable_weight_caching(&mut self)

Disable weight caching to save memory.

pub fn is_weight_cached(&self) -> bool

Check if weight caching is enabled.

pub fn config(&self) -> &QLoraConfig

Get the QLoRA configuration used to create this layer.

pub fn lora(&self) -> &LoraLayer

Get the LoRA adapter.

pub fn lora_mut(&mut self) -> &mut LoraLayer

Get mutable access to the LoRA adapter.

pub fn lora_weights(&self) -> (&Tensor, &Tensor)

Get the LoRA A and B weight tensors.

Returns (lora_a, lora_b) where:

• lora_a has shape [r, in_features]
• lora_b has shape [out_features, r]

pub fn num_trainable_parameters(&self) -> usize

Get the number of trainable parameters (LoRA only).

pub fn memory_bytes(&self) -> usize

Get total memory usage in bytes.