TabNSM: Neural Sparse Mixer for Tabular Regression
2026-08-18 • Machine Learning
Machine LearningComputational Engineering, Finance, and Science
AI summaryⓘ
The authors propose TabNSM, a new method for predicting numbers from big, complex tables of data. Their method smartly picks which features to focus on and learns from the relationships among them without getting too slow. They add special parts to improve predictions step-by-step, handle the order in the target values, and focus more on harder examples. When tested on various datasets, TabNSM performed well, especially with lots of features or mixed types of data. This shows their approach balances accuracy and speed for this tricky kind of prediction.
tabular regressionsparse attentionfeature interactionmulti-stage regressionordinal regressionsampling strategieshigh-dimensional datarepresentation learning
Authors
Ali Eslamian, Qiang Cheng
Abstract
Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.