Struct Svrg 

pub struct Svrg {
    pub lr: f64,
    pub n: usize,
    pub update_freq: usize,
    /* private fields */
}

Stochastic Variance Reduced Gradient (SVRG; Johnson & Zhang 2013).

SVRG maintains a snapshot of parameters and the corresponding full gradient, and uses a control variate to reduce variance of the stochastic gradient estimate:

g̃ = g_i(x) - g_i(x̃) + ∇f(x̃) x ← x - lr · g̃

The snapshot (x̃, ∇f(x̃)) must be updated every update_freq inner steps by calling Svrg::update_snapshot.

Fields

lr: f64

Learning rate.

n: usize

Dataset size (used for normalization in documentation, not internally).

update_freq: usize

Number of inner steps between snapshot updates.

Implementations

impl Svrg

pub fn new(lr: f64, n: usize, update_freq: usize) -> Self

Create a new SVRG optimizer.

Arguments

• lr – Learning rate.
• n – Dataset size.
• update_freq – Inner-loop length (snapshot updated every this many steps).

pub fn step( &mut self, params: &mut Vec<f64>, stochastic_grad: &[f64], snapshot_grad_i: &[f64], ) -> OptimizeResult<()>

Perform one SVRG inner-loop update.

Arguments

• params – Current parameters (modified in place).
• stochastic_grad – Mini-batch gradient at current params.
• snapshot_grad_i – Mini-batch gradient at the snapshot params (same mini-batch).

Errors

Returns OptimizeError::InvalidInput on length mismatches, or if update_snapshot has not been called first.

pub fn update_snapshot(&mut self, params: &[f64], full_grad: &[f64])

Update the snapshot with current parameters and full gradient.

Should be called at the start of each outer epoch (every update_freq inner steps).

pub fn needs_snapshot_update(&self) -> bool

Whether the inner loop has completed and a snapshot update is due.

pub fn snapshot_params(&self) -> &[f64]

Current snapshot parameters.

Trait Implementations

impl Clone for Svrg

fn clone(&self) -> Svrg

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Svrg

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.