Struct AdamW 

pub struct AdamW<A: Float + ScalarOperand + Debug> { /* private fields */ }

AdamW optimizer

Implements the AdamW optimization algorithm from the paper: “Decoupled Weight Decay Regularization” by Loshchilov and Hutter (2019).

AdamW uses a more principled approach to weight decay compared to standard Adam. The key difference is that weight decay is applied directly to the weights, not within the adaptive learning rate computation.

Formula: m_t = beta1 * m_{t-1} + (1 - beta1) * g_t v_t = beta2 * v_{t-1} + (1 - beta2) * g_t^2 m_hat_t = m_t / (1 - beta1^t) v_hat_t = v_t / (1 - beta2^t) theta_t = theta_{t-1} * (1 - lr * weight_decay) - lr * m_hat_t / (sqrt(v_hat_t) + epsilon)

Note the decoupling of weight decay from the adaptive learning rate computation.

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::{AdamW, Optimizer};

// Initialize parameters and gradients
let params = Array1::zeros(5);
let gradients = Array1::from_vec(vec![0.1, 0.2, -0.3, 0.0, 0.5]);

// Create an AdamW optimizer with default hyperparameters
let mut optimizer = AdamW::new(0.001);

// Update parameters
let new_params = optimizer.step(&params, &gradients).unwrap();

Implementations

impl<A: Float + ScalarOperand + Debug + Send + Sync> AdamW<A>

pub fn new(learning_rate: A) -> Self

Creates a new AdamW optimizer with the given learning rate and default settings

Arguments

• learning_rate - The learning rate for parameter updates

pub fn new_with_config( learning_rate: A, beta1: A, beta2: A, epsilon: A, weight_decay: A, ) -> Self

Creates a new AdamW optimizer with the full configuration

Arguments

• learning_rate - The learning rate for parameter updates
• beta1 - Exponential decay rate for the first moment estimates (default: 0.9)
• beta2 - Exponential decay rate for the second moment estimates (default: 0.999)
• epsilon - Small constant for numerical stability (default: 1e-8)
• weight_decay - Weight decay factor (default: 0.01)

pub fn set_beta1(&mut self, beta1: A) -> &mut Self

Sets the beta1 parameter

pub fn get_beta1(&self) -> A

Gets the beta1 parameter

pub fn set_beta2(&mut self, beta2: A) -> &mut Self

Sets the beta2 parameter

pub fn get_beta2(&self) -> A

Gets the beta2 parameter

pub fn set_epsilon(&mut self, epsilon: A) -> &mut Self

Sets the epsilon parameter

pub fn get_epsilon(&self) -> A

Gets the epsilon parameter

pub fn set_weight_decay(&mut self, weight_decay: A) -> &mut Self

Sets the weight decay parameter

pub fn get_weight_decay(&self) -> A

Gets the weight decay parameter

pub fn learning_rate(&self) -> A

Gets the current learning rate

pub fn set_lr(&mut self, lr: A)

Sets the learning rate

pub fn reset(&mut self)

Resets the internal state of the optimizer

Trait Implementations

impl<A: Clone + Float + ScalarOperand + Debug> Clone for AdamW<A>

fn clone(&self) -> AdamW<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Float + ScalarOperand + Debug> Debug for AdamW<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A, D> Optimizer<A, D> for AdamW<A>

where A: Float + ScalarOperand + Debug + Send + Sync, D: Dimension,

fn step( &mut self, params: &Array<A, D>, gradients: &Array<A, D>, ) -> Result<Array<A, D>>

Updates parameters using the given gradients

fn get_learning_rate(&self) -> A

Gets the current learning rate

fn set_learning_rate(&mut self, learning_rate: A)

Sets a new learning rate

fn step_list( &mut self, params_list: &[&Array<A, D>], gradients_list: &[&Array<A, D>], ) -> Result<Vec<Array<A, D>>>

Updates multiple parameter arrays at once