Struct dfdx::optim::Adam

pub struct Adam {
    pub lr: f32,
    pub betas: [f32; 2],
    pub eps: f32,
    /* private fields */
}

An implementation of the Adam optimizer from Adam: A Method for Stochastic Optimization

Example Usage:

let mut t = Tensor0D::ones();
let mut opt: Adam = Default::default();
let gradients = t.trace().backward();
opt.update(&mut t, gradients);

Changing default parmeters:

let adam = Adam::new(1e-2, [0.5, 0.25], 1e-6);

Fields

lr: f32

Learning rate

betas: [f32; 2]

Betas from Adam paper

eps: f32

Epsilon for numerical stability

Implementations

impl Adam

pub fn new(lr: f32, betas: [f32; 2], epsilon: f32) -> Self

Construct with control over all fields.

Trait Implementations

impl Debug for Adam

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Adam

Use the default parameters suggested in the paper of lr=1e-3, beta1=0.9, beta2=0.999, and epsilon=1e-8

fn default() -> Self

• Self::lr 1e-3
• Self::betas [0.9, 0.999]
• Self::eps 1e-8

impl GradientProvider for Adam

fn gradient<P>(&mut self, p: &P) -> Box<P::Array> where
    P: HasUniqueId + HasArrayType<Dtype = f32> + HasDevice, 

Retrieves the data associated with p if there is any. This can modify self, for instance if velocities are calculated based on the associated data!

impl Optimizer for Adam

fn update<M: CanUpdateWithGradients>(
    &mut self,
    module: &mut M,
    gradients: Gradients
)