Skip to main content

Crate burn_muon_plus

Crate burn_muon_plus 

Source
Expand description

§burn-muon-plus — Muon+ optimizer for Burn

arXivComponentWhat
2602.21545Muon+Muon + one post-polar normalization step (up to 37% pre-training speedup, zero extra state)

§Muon+

Newton–Schulz polar iterations flatten the singular spectrum of the momentum matrix, but in practice they amplify column/row norm imbalance in the update (“post-polar imbalanced update problem”, 2602.21545 §2.1). This imbalance tightens the second-order term in a blockwise descent analysis and shrinks Muon’s largest stable step size.

Muon+ inserts a single normalization step after orthogonalization:

M_t  = μ·M_{t-1} + (1-μ)·G_t
O_t  = Norm_(d)(Ortho(M_t))      # NS polar + row/col normalization
W_t  = W_{t-1} - η·√(m/n)·O_t
  • Norm_col(X) = X·D_col⁻¹ (unit-L2 columns)
  • Norm_row(X) = D_row⁻¹·X (unit-L2 rows)
  • ColRow/RowCol = sequential composition (best in the paper)

Non-matrix parameters (biases, norms, embeddings) use AdamW — the paper’s recipe (§3.1), matching Muon’s convention.

§Example

ⓘ
use burn_muon_plus::{MuonPlusConfig, NormDir};

let mut optimizer = MuonPlusConfig::new()
    .with_norm_dir(NormDir::ColRow)   // paper's best
    .init();
let model = optimizer.step(lr, model, grads);

Structs§

MuonPlus
Hybrid Muon+ (2D) + AdamW (1D) optimizer.
MuonPlusConfig
Configuration for the hybrid Muon+ (2D) + AdamW (1D) optimizer.
MuonPlusState
State of the hybrid optimizer for one parameter.
MuonPlusStateItem
The record item type for the module.

Enums§

NormDir
Muon+ post-polar normalization direction (2602.21545).

Constants§

NS_COEFFS
Newton-Schulz quintic coefficients (Keller Jordan, 2602.21545 uses them).