Blog: Survey of Optimizers
2026-08-28 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explain that by 2025-2026, neural network optimization is more complex than just small tweaks to the Adam algorithm. They categorize new optimization methods along four dimensions: how estimates change over time, the shape of updates, how training schedules are managed, and how optimizers handle memory and computation constraints. They note that while matrix-based methods are a real improvement, there is still no one-size-fits-all replacement for AdamW, as the best choice depends on factors like model size and training conditions. Their work encourages thinking of optimizers as combinations of features rather than single solutions and calls for better ways to compare them.
AdamWmatrix-aware optimizationtemporal estimationupdate geometryhorizon managementrepresentation and systemsspectral normalizationbatch sizequantized optimizer statestraining schedules
Authors
Ruoran Xu
Abstract
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.