Struct Sophia 

pub struct Sophia {
    pub lr: f32,
    pub beta1: f32,
    pub beta2: f32,
    pub gamma: f32,
    pub rho: f32,
    pub eps: f32,
    pub weight_decay: f32,
    /* private fields */
}

Sophia-H — Hessian-diagonal second-order optimizer.

Fields

lr: f32

Learning rate. Typically slightly larger than the AdamW LR you’d use on the same model, because the clip bounds the step.

beta1: f32

First-moment EMA decay β₁. Default 0.965.

beta2: f32

Hessian-diagonal EMA decay β₂. Default 0.99.

gamma: f32

Hessian scale γ (Liu et al. default 0.01). Multiplies the Hessian estimate before forming the denominator.

rho: f32

Per-coordinate clip threshold ρ. Default 0.04 — the dimensionless cap on each step’s magnitude.

eps: f32

Denominator floor. Default 1e-12.

weight_decay: f32

Decoupled weight-decay coefficient λ. Default 0.1 (large by AdamW standards — Sophia tolerates more decay).

Implementations

impl Sophia

pub fn new(lr: f32) -> Self

Construct with (β₁, β₂, γ, ρ, ε, λ) = (0.965, 0.99, 0.01, 0.04, 1e-12, 0.1).

pub fn with_betas(self, b1: f32, b2: f32) -> Self

Override (β₁, β₂).

pub fn with_weight_decay(self, wd: f32) -> Self

Override the decoupled-decay coefficient.

pub fn update_hessian(&mut self, name: &str, h_hat: &[f32])

Update the diagonal-Hessian estimate for parameter name. h_hat should be a fresh estimate (typically H_diag from a Hutchinson estimator or g² from a Gauss-Newton approximation).

Trait Implementations

impl Clone for Sophia

fn clone(&self) -> Sophia

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Sophia

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Optimizer for Sophia

fn step( &mut self, name: &str, _shape: &[usize], param: &mut [f32], grad: &[f32], )

fn end_iteration(&mut self)

Advance the global step counter. Most algorithms increment per call to [step], so most implementations leave this a no-op.

fn lr_scale(&self, _name: &str) -> f32

Per-tensor multiplier on the effective learning rate. Default is 1.0 for every name. Override when wrapping this crate to support per-name LR schedules (e.g. embedding-vs-attention splits, or the Gaussian-splat attribute-typed LR setup). The CPU impls in this crate currently honor this only when the caller passes a pre-scaled lr for the relevant call — backends are encouraged to consult it inside their fused kernel.