Module adaptive_optimizer 

AdaptiveOptimizer — Adam, AdaGrad, RMSProp, and AdamW optimizers for distributed gradient descent.

Overview

This module implements four widely-used adaptive gradient optimizers:

• Adam — first- and second-moment estimates with bias correction.
• AdaGrad — cumulative squared-gradient denominator, good for sparse gradients.
• RMSProp — exponentially-decaying squared-gradient estimate with optional momentum.
• AdamW — Adam with decoupled weight-decay regularisation (Loshchilov & Hutter 2019).

All optimizers share a common AdaptiveOptimizer driver that manages per-group OptimizerState lazily and exposes helpers for gradient clipping, norm computation, and statistics.

Structs

AdaptiveOptimizer

Adaptive gradient optimizer supporting Adam, AdaGrad, RMSProp, and AdamW.

OptimizerState

Per-parameter-group optimizer state (first moment, second moment, step counter).

OptimizerStats

A lightweight statistics snapshot returned by AdaptiveOptimizer::stats.

ParameterGroup

A named group of parameters together with their current gradients.

Enums

OptimizerAlgorithm

Choice of adaptive gradient algorithm and its hyper-parameters.

OptimizerError

Errors produced by the adaptive optimizer.