Module dfdx::optim

Optimizers such as Sgd, Adam, and RMSprop that can optimize neural networks.

Initializing

All the optimizer’s provide Default implementations, and also provide a way to specify all the relevant parameters through the corresponding config object:

• Sgd::new() with SgdConfig
• Adam::new() with AdamConfig
• RMSprop::new() with RMSpropConfig

Updating network parameters

This is done via Optimizer::update(), where you pass in a mutable crate::nn::Module, and the crate::gradients::Gradients:

let mut model: MyModel = Default::default();
let mut opt: Sgd<MyModel> = Default::default();
// -- snip loss computation --

let gradients: Gradients = loss.backward();
opt.update(&mut model, gradients);

Structs

Adam

An implementation of the Adam optimizer from Adam: A Method for Stochastic Optimization

AdamConfig

Configuration of hyperparameters for Adam.

RMSprop

RMSprop As described in Hinton, 2012.

RMSpropConfig

Configuration of hyperparameters for RMSprop.

Sgd

Implementation of Stochastic Gradient Descent. Based on pytorch’s implementation

SgdConfig

Configuration of hyperparameters for Sgd.

Enums

Momentum

Momentum used for Sgd

Traits

Optimizer

All optimizers must implement the update function, which takes an object that implements CanUpdateWithGradients, and calls CanUpdateWithGradients::update.