Struct RAdam 

pub struct RAdam<A: Float + ScalarOperand + Debug> { /* private fields */ }

RAdam (Rectified Adam) optimizer

Implements the RAdam algorithm from the paper: “On the Variance of the Adaptive Learning Rate and Beyond” by Liu et al. (2019).

RAdam improves upon Adam by addressing the early-stage training instability with a rectified variance term. It eliminates the need for a warmup period and often leads to better convergence.

Formula: m_t = beta1 * m_{t-1} + (1 - beta1) * g_t v_t = beta2 * v_{t-1} + (1 - beta2) * g_t^2 m_hat_t = m_t / (1 - beta1^t) v_hat_t = v_t / (1 - beta2^t)

rho_inf = 2 / (1 - beta2) - 1 rho_t = rho_inf - 2 * t * beta2^t / (1 - beta2^t)

If rho_t > 4 (the variance of the adaptive learning rate is tractable): r_t = sqrt( ((rho_t - 4)(rho_t - 2) rho_inf) / ((rho_inf - 4)(rho_inf - 2) rho_t) ) theta_t = theta_{t-1} - lr * r_t * m_hat_t / (sqrt(v_hat_t) + epsilon) Else: theta_t = theta_{t-1} - lr * m_hat_t (non-adaptive, SGD-with-momentum-like)

The rectification term r_t tends to 1 as t -> infinity, so late training behaves like Adam. See Liu et al. (2019), Algorithm 2.

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::{RAdam, Optimizer};

// Initialize parameters and gradients
let params = Array1::zeros(5);
let gradients = Array1::from_vec(vec![0.1, 0.2, -0.3, 0.0, 0.5]);

// Create a RAdam optimizer with default hyperparameters
let mut optimizer = RAdam::new(0.001);

// Update parameters
let new_params = optimizer.step(&params, &gradients).expect("optimizer.step succeeds");

Implementations

impl<A: Float + ScalarOperand + Debug + Send + Sync> RAdam<A>

pub fn new(learning_rate: A) -> Self

Creates a new RAdam optimizer with the given learning rate and default settings

Arguments

• learning_rate - The learning rate for parameter updates

pub fn new_with_config( learning_rate: A, beta1: A, beta2: A, epsilon: A, weight_decay: A, ) -> Self

Creates a new RAdam optimizer with the full configuration

Arguments

• learning_rate - The learning rate for parameter updates
• beta1 - Exponential decay rate for the first moment estimates (default: 0.9)
• beta2 - Exponential decay rate for the second moment estimates (default: 0.999)
• epsilon - Small constant for numerical stability (default: 1e-8)
• weight_decay - Weight decay factor (default: 0.0)

pub fn set_beta1(&mut self, beta1: A) -> &mut Self

Sets the beta1 parameter

pub fn get_beta1(&self) -> A

Gets the beta1 parameter

pub fn set_beta2(&mut self, beta2: A) -> &mut Self

Sets the beta2 parameter

pub fn get_beta2(&self) -> A

Gets the beta2 parameter

pub fn set_epsilon(&mut self, epsilon: A) -> &mut Self

Sets the epsilon parameter

pub fn get_epsilon(&self) -> A

Gets the epsilon parameter

pub fn set_weight_decay(&mut self, weight_decay: A) -> &mut Self

Sets the weight decay parameter

pub fn get_weight_decay(&self) -> A

Gets the weight decay parameter

pub fn learning_rate(&self) -> A

Gets the current learning rate

pub fn set_lr(&mut self, lr: A)

Sets the learning rate

pub fn reset(&mut self)

Resets the internal state of the optimizer

pub fn timestep(&self, index: usize) -> usize

Returns the timestep recorded for the parameter tensor at index

Returns 0 when the index has never been stepped.

pub fn rho_inf(&self) -> A

Returns rho_infinity, the maximum length of the approximated SMA

pub fn rho_t(&self, t: usize) -> Option<A>

Computes rho_t, the length of the approximated simple moving average at step t

Returns None when t == 0 (no step has been taken yet).

pub fn rectification_term(&self, t: usize) -> Option<A>

Computes the RAdam rectification term r_t for step t

Returns None when the variance is not yet tractable (rho_t <= 4), in which case RAdam falls back to a non-adaptive, SGD-like update.

r_t converges to 1 as t -> infinity.

pub fn step_inplace_indexed<D: Dimension>( &mut self, index: usize, params: &mut Array<A, D>, gradients: &Array<A, D>, ) -> Result<()>

Applies a RAdam update in place for the parameter tensor at index

pub fn step_inplace<D: Dimension>( &mut self, params: &mut Array<A, D>, gradients: &Array<A, D>, ) -> Result<()>

Applies a RAdam update in place using the state slot of the first parameter tensor

pub fn step_indexed<D: Dimension>( &mut self, index: usize, params: &Array<A, D>, gradients: &Array<A, D>, ) -> Result<Array<A, D>>

Performs a RAdam update for the parameter tensor at index

Each index owns an independent moment/timestep slot.

Trait Implementations

impl<A: Clone + Float + ScalarOperand + Debug> Clone for RAdam<A>

fn clone(&self) -> RAdam<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Float + ScalarOperand + Debug> Debug for RAdam<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A, D> Optimizer<A, D> for RAdam<A>

where A: Float + ScalarOperand + Debug + Send + Sync + From<f64>, D: Dimension,

fn step( &mut self, params: &Array<A, D>, gradients: &Array<A, D>, ) -> Result<Array<A, D>>

Updates parameters using the given gradients

fn step_list( &mut self, params_list: &[&Array<A, D>], gradients_list: &[&Array<A, D>], ) -> Result<Vec<Array<A, D>>>

Updates multiple parameter arrays at once

fn get_learning_rate(&self) -> A

Gets the current learning rate

fn set_learning_rate(&mut self, learning_rate: A)

Sets a new learning rate