Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks
2026-09-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a problem in training Physics-Informed Neural Networks (PINNs) where different parts of the training objective cause conflicting gradient directions. While existing methods fix these conflicts before the optimizer step, the authors find that the optimizer itself can reintroduce conflicts after processing the gradients, a problem they call Gradient-Update Mismatch (GUM). They propose a new method called Gradient-Update Alignment (GUA) that adjusts the optimizer’s update to keep it conflict-free. Their experiments show that GUM happens often and that GUA improves training accuracy by making updates better aligned with the original conflict-free directions.
Physics-Informed Neural Networksgradientsgradient surgeryoptimizerconflicting gradientsadaptive scalingmomentumpreconditioninggradient-update mismatchgradient-update alignment
Authors
Jing Xiao, Xinhai Chen, Qinglin Wang, Menghan Jia, Zhiquan Lai, Dongsheng Li, Jie Liu, Tiejun Li
Abstract
Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let $a_t$ denote the direction constructed by gradient surgery, $u_t$ the optimizer proposal, and $\mathcal{C}_t$ the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform $a_t$ through mechanisms such as historical state, adaptive scaling, preconditioning, or decoupled weight decay, so $a_t \in \mathcal{C}_t$ does not generally imply $u_t \in \mathcal{C}_t$. We refer to this optimizer-induced discrepancy in conflict-freeness between $a_t$ and $u_t$ as Gradient-Update Mismatch (GUM). Accordingly, we propose Gradient-Update Alignment (GUA), which projects $u_t$ onto $\mathcal{C}_t$ to obtain the aligned update $p_t$ and applies $p_t$ to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update. We conduct extensive experiments and find that GUM is widespread across momentum, adaptive, and curvature-based optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing the relative $L_2$ error by up to 98.2% in individual settings. Data and code are available at https://github.com/JingXiao10/GUA.