AI summaryⓘ
The authors studied how a model called Muon learns modular addition and found that although it can quickly figure out the training examples (grok), its ability to generalize to new ones later fails repeatedly. This failure happens where the model’s internal representation connects to the output, and freezing certain parts of the model can prevent this collapse. They also showed that removing some normalization steps doesn’t fix the problem and analyzed the failure using Fourier methods, which distinguished between outright failure and situations where the model’s correct part is overshadowed by an adversarial one. Overall, the results highlight that subtle dynamics at the representation-output interface cause the loss of generalization after initial learning.
modular additiongrokkinggeneralizationrepresentation-readout interfaceAdamW optimizernormalizationorthogonalizationFourier filteringmodel collapse
Authors
Ali Janati, Kaoutar El Maghraoui, Andrei Kanavalau, Anass Belfatmi
Abstract
Under the standard split, Muon gets hidden matrices and AdamW embeddings/output head. Muon groks modular addition faster, but its solutions do not hold. All nine configurations on $(a+b) \bmod 113$ grok and later lose generalization. Across five seeds the selected AdamW reference falls below threshold on four, reaching 27.59%. Instability persists across two moduli, two widths, two training fractions, subtraction, and depth. The failure arises at the representation-readout interface, identified only jointly up to an invertible map unselected by the loss. After solving the training set, the gradient falls to order $10^{-6}$ and the optimizers respond differently: step-size elasticity is -0.03 for Muon versus +1.5 for AdamW, and the Muon group moves 8.0 times faster per parameter. From bit-identical states, freezing either group prevents failure. Freezing embeddings/readout removes it in five runs over 451,400 post-grokking steps and five paired seeds: unfrozen arms record 137-321 sub-threshold evaluations, frozen arms none. Removing Muon's normalization and orthogonalization is no substitute: it collapses representation from 326 effective conjugate pairs to 4, shows no recurrent collapse, and fails terminally. Fourier filtering separates circuit failure from masking. Across 43 checkpoints over five seeds and three regimes, the task-aligned family reaches exactly 100% alone. In circuit failure it no longer solves the task; in masking it remains perfect while the full model reaches 45.85%, giving a positive margin on every example, including errors, but being outvoted by a near-equal adversarial remainder. Rescaling it restores 99.9%; grokking is the same condition resolving upward. The task selects the family, swapping $(k,k)$ for $(k,-k)$ under subtraction. Across an abrupt collapse, standard Fourier support is unchanged and the power-distribution cosine remains 0.9899.