GIST: Gauge-Invariant Spectral Transformers for Scalable Graph Neural Operators
2026-03-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a problem with using transformer models on graph or mesh data, where existing methods are either too slow or lose important mathematical properties called gauge invariance. They introduce GIST, a new transformer design that is faster and keeps these properties intact by using random projections and special attention mechanisms. This allows their model to learn consistently even when the graphs or meshes change resolution or structure. Their method performs as well as top models on standard graph tasks and works well on very large mesh problems related to aerodynamics.
transformerpositional encodinggraph-structured dataspectral methodsgauge invariancerandom projectionsneural operatorsmesh discretizationattention mechanisminductive learning
Authors
Mattia Rigotti, Nicholas Thumiger, Thomas Frick
Abstract
Adapting transformer positional encoding to meshes and graph-structured data presents significant computational challenges: exact spectral methods require cubic-complexity eigendecomposition and can inadvertently break gauge invariance through numerical solver artifacts, while efficient approximate methods sacrifice gauge symmetry by design. Both failure modes cause catastrophic generalization in inductive learning, where models trained with one set of numerical choices fail when encountering different spectral decompositions of similar graphs or discretizations of the same mesh. We propose GIST (Gauge-Invariant Spectral Transformers), a new graph transformer architecture that resolves this challenge by achieving end-to-end $\mathcal{O}(N)$ complexity through random projections while algorithmically preserving gauge invariance via inner-product-based attention on the projected embeddings. We prove GIST achieves discretization-invariant learning with bounded mismatch error, enabling parameter transfer across arbitrary mesh resolutions for neural operator applications. Empirically, GIST matches state-of-the-art on standard graph benchmarks (e.g., achieving 99.50% micro-F1 on PPI) while uniquely scaling to mesh-based Neural Operator benchmarks with up to 750K nodes, achieving state-of-the-art aerodynamic prediction on the challenging DrivAerNet and DrivAerNet++ datasets.