Enum Decay 

pub enum Decay {
    WeightDecay(f64),
    DecoupledWeightDecay(f64),
}

Method of weight decay to use

Variants

WeightDecay(f64)

Weight decay regularisation to penalise large weights

The gradient is transformed as $$ g_{t} \gets g_{t} + \lambda \theta_{t-1}$$

This is equivalent to an L2 regularisation term in the loss adding $\frac{\lambda}{2}||\theta||_{2}^{2}$ but avoids autodifferentiation of the L2 term

DecoupledWeightDecay(f64)

Decoupled weight decay as described in Decoupled Weight Decay Regularization

This directly decays the weights as

$$ \theta_{t} \gets (1 - \eta \lambda) \theta_{t-1}$$

This is equivalent to regularisation, only for SGD without momentum, but is different for adaptive gradient methods

Trait Implementations

impl Clone for Decay

fn clone(&self) -> Decay

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Decay

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Decay

fn eq(&self, other: &Decay) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for Decay

fn partial_cmp(&self, other: &Decay) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Copy for Decay

impl StructuralPartialEq for Decay