Struct SparseLinear 

pub struct SparseLinear {
    pub weight: Parameter,
    pub bias: Option<Parameter>,
    pub threshold: Parameter,
    /* private fields */
}

A linear layer with a differentiable magnitude pruning mask.

During the forward pass, a soft mask is computed via sigmoid soft thresholding:

mask = sigmoid((|weight| - threshold) * temperature)
effective_weight = weight * mask
y = x @ effective_weight^T + bias

The sigmoid makes the mask differentiable, so gradients flow through it and the network learns which weights to prune. The threshold parameter is learnable and included in parameters().

Structured vs Unstructured

• Structured (structured=true): One threshold per output neuron. Entire output channels can be pruned, yielding hardware-friendly sparsity.
• Unstructured (structured=false): One threshold per weight element. Finer-grained but less hardware-friendly.

Example

let layer = SparseLinear::new(784, 256);
let output = layer.forward(&input);
println!("Density: {:.1}%", layer.density() * 100.0);

Fields

weight: Parameter

Weight matrix of shape (out_features, in_features).

bias: Option<Parameter>

Optional bias vector of shape (out_features).

threshold: Parameter

Learnable magnitude thresholds. Shape depends on structured:

• Structured: (out_features,)
• Unstructured: (out_features, in_features)

Implementations

impl SparseLinear

pub fn new(in_features: usize, out_features: usize) -> Self

Creates a new SparseLinear layer with structured pruning and bias.

Arguments

• in_features - Size of each input sample
• out_features - Size of each output sample

pub fn unstructured(in_features: usize, out_features: usize) -> Self

Creates a new SparseLinear layer with unstructured (per-weight) pruning.

Arguments

• in_features - Size of each input sample
• out_features - Size of each output sample

pub fn with_bias(in_features: usize, out_features: usize, bias: bool) -> Self

Creates a new SparseLinear layer with configurable bias.

Arguments

• in_features - Size of each input sample
• out_features - Size of each output sample
• bias - Whether to include a learnable bias

pub fn in_features(&self) -> usize

Returns the input feature dimension.

pub fn out_features(&self) -> usize

Returns the output feature dimension.

pub fn is_structured(&self) -> bool

Returns whether this layer uses structured pruning.

pub fn density(&self) -> f32

Returns the fraction of weights that are active (above threshold).

Uses hard thresholding at |weight| >= threshold.

pub fn sparsity(&self) -> f32

Returns the fraction of weights that are pruned.

Equivalent to 1.0 - density().

pub fn num_active(&self) -> usize

Returns the number of active (non-pruned) weights.

pub fn hard_prune(&mut self)

Permanently applies the pruning mask to the weights.

After calling this, pruned weights are zeroed and the threshold is reset to zero. This is an irreversible optimization for inference — the zeroed weights will not be recovered.

pub fn reset_threshold(&mut self, value: f32)

Resets the threshold to a specific value.

Arguments

• value - The new threshold value

pub fn effective_weight(&self) -> Tensor<f32>

Returns the effective weight (weight * hard_mask) for inspection.

This shows what the weight matrix looks like after hard pruning, without actually modifying the layer.

Trait Implementations

impl Debug for SparseLinear

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Module for SparseLinear

fn forward(&self, input: &Variable) -> Variable

Performs the forward pass.

fn parameters(&self) -> Vec<Parameter>

Returns all parameters of this module.

fn named_parameters(&self) -> HashMap<String, Parameter>

Returns named parameters of this module.

fn name(&self) -> &'static str

Returns the module name for debugging.

fn num_parameters(&self) -> usize

Returns the number of trainable parameters.

fn train(&mut self)

Sets the module to training mode.

fn eval(&mut self)

Sets the module to evaluation mode.

fn set_training(&mut self, _training: bool)

Sets the training mode.

fn is_training(&self) -> bool

Returns whether the module is in training mode.

fn zero_grad(&self)

Zeros all gradients of parameters.

fn to_device(&self, device: Device)

Moves all parameters to the specified device.