Struct LAMB 

pub struct LAMB { /* private fields */ }

LAMB optimizer for large batch training.

LAMB extends Adam by adding a layer-wise trust ratio that scales the update based on the ratio of parameter norm to update norm. This enables stable training with very large batch sizes.

The update rule is:

m_t = beta1 * m_{t-1} + (1 - beta1) * grad
v_t = beta2 * v_{t-1} + (1 - beta2) * grad^2
m_hat = m_t / (1 - beta1^t)
v_hat = v_t / (1 - beta2^t)
r = m_hat / (sqrt(v_hat) + eps) + weight_decay * param
trust_ratio = ||param|| / ||r||  (layer-wise)
param = param - lr * trust_ratio * r

Implementations

impl LAMB

pub fn new(params: Vec<Parameter>, lr: f32) -> Self

Creates a new LAMB optimizer with default hyperparameters.

Defaults:

• betas: (0.9, 0.999)
• eps: 1e-6
• weight_decay: 0.0

pub fn with_betas(params: Vec<Parameter>, lr: f32, betas: (f32, f32)) -> Self

Creates LAMB with specified betas.

pub fn with_options( params: Vec<Parameter>, lr: f32, betas: (f32, f32), eps: f32, weight_decay: f32, ) -> Self

Creates LAMB with all options.

pub fn betas(self, beta1: f32, beta2: f32) -> Self

Builder: set betas (momentum decay rates)

pub fn eps(self, eps: f32) -> Self

Builder: set epsilon

pub fn weight_decay(self, weight_decay: f32) -> Self

Builder: set weight decay

pub fn bias_correction(self, enabled: bool) -> Self

Builder: set bias correction

Trait Implementations

impl Optimizer for LAMB

fn step(&mut self)

Performs a single optimization step.

fn zero_grad(&mut self)

Zeros all parameter gradients.

fn get_lr(&self) -> f32

Returns the current learning rate.

fn set_lr(&mut self, lr: f32)

Sets the learning rate.

fn parameters(&self) -> &[Parameter]

Returns the parameters being optimized.

fn num_parameters(&self) -> usize

Returns the number of parameters.