Module dfdx::optim

Optimizers such as Sgd, Adam, and RMSprop that can optimize neural networks.

Initializing

All the optimizer’s provide Default implementations, and also provide a way to specify all the relevant parameters through the corresponding config object:

• Sgd::new() with SgdConfig
• Adam::new() with AdamConfig
• RMSprop::new() with RMSpropConfig

Updating network parameters

This is done via Optimizer::update(), where you pass in a mutable crate::nn::Module, and the crate::tensor::Gradients:

let mut model = MyModel::build_on_device(&dev);
let mut grads = model.alloc_grads();
let mut opt = Sgd::new(&model, Default::default());
// -- snip loss computation --

grads = loss.backward();
opt.update(&mut model, &grads);
model.zero_grads(&mut grads);

Re-exports

• pub use crate::tensor_ops::AdamConfig;
• pub use crate::tensor_ops::Momentum;
• pub use crate::tensor_ops::RMSpropConfig;
• pub use crate::tensor_ops::SgdConfig;
• pub use crate::tensor_ops::WeightDecay;

Modules

• prelude

Structs

• Adam
  An implementation of the Adam optimizer from Adam: A Method for Stochastic Optimization
• RMSprop
  RMSprop As described in Hinton, 2012.
• Sgd
  Implementation of Stochastic Gradient Descent. Based on pytorch’s implementation
• UnusedTensors
  Holds UniqueId of tensors that were missing gradients during update, and therefore are unused

Enums

• OptimizerUpdateError
  An error indicating that a parameter was not used in gradient computation, and was therefore not present in Gradients during an update.

Traits

• Optimizer
  All optimizers must implement the update function, which takes a M and updates all of its parameters.