Expand description
§burn-muon-plus — Muon+ optimizer for Burn
| arXiv | Component | What |
|---|---|---|
| 2602.21545 | Muon+ | Muon + one post-polar normalization step (up to 37% pre-training speedup, zero extra state) |
§Muon+
Newton–Schulz polar iterations flatten the singular spectrum of the momentum matrix, but in practice they amplify column/row norm imbalance in the update (“post-polar imbalanced update problem”, 2602.21545 §2.1). This imbalance tightens the second-order term in a blockwise descent analysis and shrinks Muon’s largest stable step size.
Muon+ inserts a single normalization step after orthogonalization:
M_t = μ·M_{t-1} + (1-μ)·G_t
O_t = Norm_(d)(Ortho(M_t)) # NS polar + row/col normalization
W_t = W_{t-1} - η·√(m/n)·O_tNorm_col(X) = X·D_col⁻¹(unit-L2 columns)Norm_row(X) = D_row⁻¹·X(unit-L2 rows)ColRow/RowCol= sequential composition (best in the paper)
Non-matrix parameters (biases, norms, embeddings) use AdamW — the paper’s recipe (§3.1), matching Muon’s convention.
§Example
ⓘ
use burn_muon_plus::{MuonPlusConfig, NormDir};
let mut optimizer = MuonPlusConfig::new()
.with_norm_dir(NormDir::ColRow) // paper's best
.init();
let model = optimizer.step(lr, model, grads);Structs§
- Muon
Plus - Hybrid Muon+ (2D) + AdamW (1D) optimizer.
- Muon
Plus Config - Configuration for the hybrid Muon+ (2D) + AdamW (1D) optimizer.
- Muon
Plus State - State of the hybrid optimizer for one parameter.
- Muon
Plus State Item - The record item type for the module.
Enums§
- NormDir
- Muon+ post-polar normalization direction (2602.21545).
Constants§
- NS_
COEFFS - Newton-Schulz quintic coefficients (Keller Jordan, 2602.21545 uses them).