Struct drug::Optimizer

pub struct Optimizer {
    pub learning_rate: f32,
    pub beta_momentum: f32,
    pub beta_magnitude: f32,
    pub epsilon: f32,
    /* private fields */
}

Here is a good blog that explains various optimizers. Currently only SGD, RMSProp, Adam, and SGD-with-momentum are implemented. The Optimizerstruct builds and holds OptimizerInstances which hold runtime information about every parameter that’s being optimized. If beta_momentum or beta_magnitude are set to zero, then the optimizer does not keep momentum and magnitude correction information information about parameters. epsilon is added to denominators to avoid divide by zero errors.

	no beta_momentum	beta_momentum
no beta_magnitude	vanilla SGD	SGD with momentum
beta_magnitude	RMSProp	Adam

Fields

learning_rate: f32

beta_momentum: f32

beta_magnitude: f32

epsilon: f32

Implementations

impl Optimizer

pub fn new(
    learning_rate: f32,
    beta_momentum: f32,
    beta_magnitude: f32,
    epsilon: f32
) -> Self

pub fn sgd_default() -> Self

Vanilla stochastic gradient descent with no added fluff.

pub fn momentum_default() -> Self

SGD with a momentum component. Add the geometric average of past gradients to the parameter instead of the gradient itself. This averaging dampens the stochasticity of the stochastic gradient descent.

pub fn rmsprop_default() -> Self

SGD with a magnitude component. Rescale gradients by dividing by the geometric mean of previous gradients squared. Parameters with frequent large gradients will see those gradients shrink while parameters with sparse gradients will have their gradients grow.

pub fn adam_default() -> Self

Adam (Adaptive Moment Estimation) Combines the momentum component from momentum and the magnitude component from rmsprop.

pub fn register(&mut self, i: Idx, shape: &[usize])

pub fn apply_gradient(
    &mut self,
    i: Idx,
    param: ArrayViewMutD<'_, f32>,
    grad: &ArrayD<f32>
)

Apply gradient

Trait Implementations

impl Debug for Optimizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Optimizer

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Optimizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
    __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for Optimizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Serialize for Optimizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>where
    __S: Serializer,

Serialize this value into the given Serde serializer.