Struct dfdx::optim::sgd::Sgd

pub struct Sgd {
    pub lr: f32,
    pub momentum: Option<Momentum>,
    /* private fields */
}

Implementation of Stochastic Gradient Descent. Based on pytorch’s implementation

Nesterov Momentum is implemented as described in On the importance of initialization and momentum in deep learning.

Example Usage:

use dfdx::prelude::*;

let mut t = Tensor0D::ones();
let mut opt: Sgd = Default::default();

let gradients = t.trace().backward();
opt.update(&mut t, gradients);

Changing default parmeters:

use dfdx::optim::{Sgd, Momentum};

let sgd_no_momentum = Sgd::new(1e-1, None);
let sgd_classic_momentum = Sgd::new(1e-2, Some(Momentum::Classic(0.5)));
let sgd_nesterov_momentum = Sgd::new(1e-3, Some(Momentum::Nesterov(0.25)));

Fields

lr: f32

momentum: Option<Momentum>

Implementations

impl Sgd

pub fn new(lr: f32, momentum: Option<Momentum>) -> Self

Trait Implementations

impl Debug for Sgd

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Sgd

fn default() -> Self

Returns the “default value” for a type.

impl GradientProvider for Sgd

fn gradient<P>(&mut self, p: &P) -> Option<Box<P::Array>> where
    P: HasUniqueId + HasArrayType<Dtype = f32> + HasDevice, 

Retrieves the data associated with p if there is any. This can modify self, for instance if velocities are calculated based on the associated data!

impl Optimizer for Sgd

fn update<M: CanUpdateWithGradients>(
    &mut self,
    module: &mut M,
    gradients: Gradients
)