Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
2026-08-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors found that during language model training, the ratio between learning rate and parameter norm—called the effective learning rate (ELR)—controls how the loss changes. When this ELR is the same, different runs with various learning rates and norms show almost identical loss patterns. They tested this across many settings and discovered that factors like normalization and how quickly learning rate and norms change impact how well these patterns match. Their work also shows that common methods like weight decay mainly influence training through their effect on ELR. Overall, the authors propose ELR as a key concept connecting learning rate schedules, norm control, and how loss behaves during training.
Effective Learning Rate (ELR)Learning Rate (LR)Parameter NormLoss DynamicsNormalizationWeight DecayLanguage Model PretrainingOptimizerFunctional Scaling Law (FSL)Norm Control
Authors
Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, Lei Wu
Abstract
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.