Struct SparseAdam 

pub struct SparseAdam<A: Float + ScalarOperand + Debug> { /* private fields */ }

SparseAdam optimizer for sparse gradients

Implements a variant of the Adam optimization algorithm that’s optimized for sparse gradients. It only updates the parameters and momentum vectors for indices that have non-zero gradients, saving computation and memory.

This optimizer is particularly useful for large embedding layers or models with sparse input features.

Formula (for non-zero gradient indices): m_t = beta1 * m_{t-1} + (1 - beta1) * g_t v_t = beta2 * v_{t-1} + (1 - beta2) * g_t^2 m_hat_t = m_t / (1 - beta1^t) v_hat_t = v_t / (1 - beta2^t) theta_t = theta_{t-1} - alpha * m_hat_t / (sqrt(v_hat_t) + epsilon)

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::{SparseAdam, SparseGradient, Optimizer};

// Initialize parameters
let params = Array1::zeros(5);

// Create sparse gradient with non-zero values at indices 1 and 3
let sparse_grad = SparseGradient::new(
    vec![1, 3],             // Indices
    vec![0.2, 0.5],         // Values
    5                       // Total dimension
);

// Create a SparseAdam optimizer
let mut optimizer = SparseAdam::new(0.001);

// Update parameters with sparse gradient
let new_params = optimizer.step_sparse(&params, &sparse_grad).unwrap();

Implementations

impl<A: Float + ScalarOperand + Debug + Send + Sync> SparseAdam<A>

pub fn new(learning_rate: A) -> Self

Creates a new SparseAdam optimizer with the given learning rate and default settings

Arguments

• learning_rate - The learning rate for parameter updates

pub fn new_with_config( learning_rate: A, beta1: A, beta2: A, epsilon: A, weight_decay: A, ) -> Self

Creates a new SparseAdam optimizer with the full configuration

Arguments

• learning_rate - The learning rate for parameter updates
• beta1 - Exponential decay rate for the first moment estimates (default: 0.9)
• beta2 - Exponential decay rate for the second moment estimates (default: 0.999)
• epsilon - Small constant for numerical stability (default: 1e-8)
• weight_decay - Weight decay factor for L2 regularization (default: 0.0)

pub fn set_beta1(&mut self, beta1: A) -> &mut Self

Sets the beta1 parameter

pub fn with_beta1(self, beta1: A) -> Self

Builder method to set beta1 and return self

pub fn get_beta1(&self) -> A

Gets the beta1 parameter

pub fn set_beta2(&mut self, beta2: A) -> &mut Self

Sets the beta2 parameter

pub fn with_beta2(self, beta2: A) -> Self

Builder method to set beta2 and return self

pub fn get_beta2(&self) -> A

Gets the beta2 parameter

pub fn set_epsilon(&mut self, epsilon: A) -> &mut Self

Sets the epsilon parameter

pub fn with_epsilon(self, epsilon: A) -> Self

Builder method to set epsilon and return self

pub fn get_epsilon(&self) -> A

Gets the epsilon parameter

pub fn set_weight_decay(&mut self, weight_decay: A) -> &mut Self

Sets the weight decay parameter

pub fn with_weight_decay(self, weight_decay: A) -> Self

Builder method to set weight decay and return self

pub fn get_weight_decay(&self) -> A

Gets the weight decay parameter

pub fn step_sparse( &mut self, params: &Array<A, Ix1>, gradient: &SparseGradient<A>, ) -> Result<Array<A, Ix1>>

Updates parameters using sparse gradients

This method efficiently updates only the parameters corresponding to non-zero gradient entries, saving computation and memory.

Arguments

• params - The parameters to update
• gradient - The sparse gradient

Returns

The updated parameters

pub fn reset(&mut self)

Resets the internal state of the optimizer

Trait Implementations

impl<A: Clone + Float + ScalarOperand + Debug> Clone for SparseAdam<A>

fn clone(&self) -> SparseAdam<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Float + ScalarOperand + Debug> Debug for SparseAdam<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A> Optimizer<A, Dim<[usize; 1]>> for SparseAdam<A>

where A: Float + ScalarOperand + Debug + Send + Sync,

fn step( &mut self, params: &Array<A, Ix1>, gradients: &Array<A, Ix1>, ) -> Result<Array<A, Ix1>>

Updates parameters using the given gradients

fn get_learning_rate(&self) -> A

Gets the current learning rate

fn set_learning_rate(&mut self, learning_rate: A)

Sets a new learning rate

fn step_list( &mut self, params_list: &[&Array<A, D>], gradients_list: &[&Array<A, D>], ) -> Result<Vec<Array<A, D>>>

Updates multiple parameter arrays at once