Struct MetaSGD 

pub struct MetaSGD<A: Float + ScalarOperand + Debug> { /* private fields */ }

Meta-SGD optimizer with per-parameter learnable learning rates

Implements the Meta-SGD algorithm which learns per-parameter learning rates alongside the model parameters. Each parameter gets its own adaptive learning rate that is updated based on the meta-gradient.

Algorithm

For each step:

1. Initialize per-parameter learning rates alpha_i to base_lr (if first step)
2. Compute parameter update: delta_i = alpha_i * grad_i
3. Update parameters: theta_i = theta_i - delta_i
4. Update per-parameter LRs: alpha_i = alpha_i - alpha_lr * grad_i * delta_i
5. Clamp alpha_i to [1e-8, 10.0]

The per-parameter learning rates evolve over time, allowing the optimizer to automatically discover the best learning rate for each parameter dimension.

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::{MetaSGD, Optimizer};

let params = Array1::from_vec(vec![1.0, 2.0, 3.0]);
let gradients = Array1::from_vec(vec![0.1, -0.2, 0.3]);

let mut optimizer = MetaSGD::new(0.01);
let new_params = optimizer.step(&params, &gradients).expect("step failed");

Implementations

impl<A: Float + ScalarOperand + Debug> MetaSGD<A>

pub fn new(base_lr: A) -> Self

Creates a new Meta-SGD optimizer with the given base learning rate

Defaults:

• alpha_lr: 0.001
• inner_steps: 5

Arguments

• base_lr - Base learning rate for initializing per-parameter LRs

pub fn with_alpha_lr(self, lr: A) -> Self

Sets the meta-learning rate for updating per-parameter learning rates

Arguments

• lr - Learning rate for the per-parameter LR updates

pub fn with_inner_steps(self, n: usize) -> Self

Sets the number of inner adaptation steps

Arguments

• n - Number of inner steps (must be >= 1)

pub fn get_base_lr(&self) -> A

Returns the base learning rate

pub fn get_alpha_lr(&self) -> A

Returns the meta-learning rate (alpha_lr)

pub fn get_inner_steps(&self) -> usize

Returns the number of inner adaptation steps

pub fn get_step_count(&self) -> usize

Returns the number of steps taken so far

pub fn get_per_param_lr(&self) -> Option<&Array<A, IxDyn>>

Returns a reference to the current per-parameter learning rates, if initialized

pub fn reset_per_param_lr(&mut self)

Resets the per-parameter learning rates (they will be re-initialized on next step)

Trait Implementations

impl<A: Clone + Float + ScalarOperand + Debug> Clone for MetaSGD<A>

fn clone(&self) -> MetaSGD<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Float + ScalarOperand + Debug> Debug for MetaSGD<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A, D> Optimizer<A, D> for MetaSGD<A>

where A: Float + ScalarOperand + Debug, D: Dimension,

fn step( &mut self, params: &Array<A, D>, gradients: &Array<A, D>, ) -> Result<Array<A, D>>

Updates parameters using the given gradients

fn get_learning_rate(&self) -> A

Gets the current learning rate

fn set_learning_rate(&mut self, learning_rate: A)

Sets a new learning rate

fn step_list( &mut self, params_list: &[&Array<A, D>], gradients_list: &[&Array<A, D>], ) -> Result<Vec<Array<A, D>>>

Updates multiple parameter arrays at once