Struct NoamScheduler 

pub struct NoamScheduler { /* private fields */ }

Noam scheduler (Transformer learning rate schedule).

This is the learning rate schedule used in “Attention is All You Need”. It increases linearly for warmup_steps, then decays proportionally to the inverse square root of the step number.

Reference: Vaswani et al. “Attention is All You Need” (NIPS 2017)

Implementations

impl NoamScheduler

pub fn new(model_dim: usize, warmup_steps: usize, scale_factor: f64) -> Self

Create a new Noam scheduler.

Arguments

• model_dim - Model dimension (d_model), typically 512 for Transformer
• warmup_steps - Number of warmup steps, typically 4000
• scale_factor - Scaling factor, typically 1.0 or 2.0

Trait Implementations

impl Clone for NoamScheduler

fn clone(&self) -> NoamScheduler

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NoamScheduler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LrScheduler for NoamScheduler

fn step(&mut self, optimizer: &mut dyn Optimizer)

Update learning rate based on current step/epoch.

fn get_lr(&self) -> f64

Get current learning rate.

fn state_dict(&self) -> HashMap<String, f64>

Get scheduler state as a dictionary.

fn load_state_dict(&mut self, state: &HashMap<String, f64>) -> TrainResult<()>

Load scheduler state from a dictionary.