Struct Adam 

pub struct Adam<T: Float> { /* private fields */ }

Adaptive Moment Estimation (ADAM) optimizer.

Formulas

1. First Moment Estimate (m):

m_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t

2. Second Moment Estimate (v):

v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2

3. Bias-Corrected Estimates:

\hat{m}_t = \frac{m_t}{1 - \beta_1^t}

\hat{v}_t = \frac{v_t}{1 - \beta_2^t}

4. Parameter Update:

\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \cdot \hat{m}_t

where:

• $ m_t $ is the first moment estimate (mean of gradients).
• $ v_t $ is the second moment estimate (uncentered variance of gradients).
• $ g_t $ is the gradient at time step $ t $.
• $ \beta_1 $ and $ \beta_2 $ are the decay rates for the moment estimates.
• $ \alpha $ is the learning rate.
• $ \epsilon $ is a small constant to prevent division by zero.
• $ \theta_t $ are the parameters being optimized.

Implementations

impl<T: Float> Adam<T>

pub fn new(learning_rate: T, architecture: &Vec<Box<dyn Function<T>>>) -> Self

pub fn full_new( learning_rate: T, beta1: T, beta2: T, epsilon: T, architecture: &Vec<Box<dyn Function<T>>>, ) -> Self

Trait Implementations

impl<T: Float> Optimizer<T> for Adam<T>

fn step(&mut self, id: usize, weights: &mut Matrix<T>, gradients: &Matrix<T>)

fn change_learning_rate(&mut self, new_learning_rate: T)