Struct Lamb 

pub struct Lamb { /* private fields */ }

Layer-wise Adaptive Moments (LAMB) optimizer.

LAMB applies an Adam-style update, then scales it by the ratio between the parameter norm and update norm. This layer-wise adaptation is useful when training with very large batch sizes.

See: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes.

Configured by LambConfig.

Trait Implementations

impl Clone for Lamb

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Optimizer for Lamb

type State<const D: usize> = LambState<D>

The state of the optimizer for a single parameter of rank D.

fn step<const D: usize>( &self, lr: LearningRate, tensor: Tensor<D>, grad: Tensor<D>, state: Option<Self::State<D>>, ) -> (Tensor<D>, Option<Self::State<D>>)

The optimizer step is performed for one tensor at a time with its gradient and state.

fn to_device<const D: usize>( state: Self::State<D>, device: &Device, ) -> Self::State<D>

Change the device of the state.