Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
2026-07-28 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a method called Sharpness-Aware Minimization (SAM), which tries to make machine learning models less sensitive to small changes in their settings, improving their overall performance. They focus on how to measure these small changes using the matrix structure of the model's layers, rather than treating parameters individually. By combining this matrix-aware approach with a new optimization technique called Muon, they improve model accuracy on popular image recognition tests. Their experiments showed that this combination worked better than previous methods.
Sharpness-Aware MinimizationMatrix-aware optimizationMuon optimizerSpectral perturbationHidden-layer weightsAdamWSGDWViT-Small/16ResNet-50ImageNet-1K
Authors
Wenzhi Zhong, Edward Milsom, Michael Murray
Abstract
Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.