Sparse Orthogonal Regression Technique: A Spectral Framework for Equation Discovery, Approximation, and Integration
2026-08-13 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce SORT, a method that finds simple, neat ways to describe complicated systems using special building blocks called orthonormal bases, even when data is messy or incomplete. Instead of traditional approaches, SORT directly learns the important pieces from the data, helping to discover the rules (like equations) governing system behavior. It performs well compared to existing methods, especially when data is noisy or limited, and it helps scientists focus on picking good bases tailored to their specific problem. Beyond finding equations, SORT also aids in approximating complex functions and calculating tricky integrals.
Sparse regressionOrthonormal basisL1-regularizationOrdinary differential equationsSpectral methodsSystem identificationSymbolic regressionNonlinear approximation
Authors
Sabin Roman, Ljupco Todorovski, Saso Dzeroski
Abstract
We develop the Sparse Orthogonal Regression Technique (SORT), a sparse spectral framework for learning orthonormal-basis expansions from noisy and irregularly sampled data. SORT estimates expansion coefficients directly from observations using L1-regularized regression, avoiding explicit quadrature or analytic inner-product evaluation. The central application is data-driven discovery of ordinary differential equations: vector fields are represented in chosen orthogonal bases and learned as sparse coefficient expansions. This provides a complementary route to symbolic regression, grammar-based discovery, and SINDy-style sparse identification by first recovering a compact spectral representation, which can later guide searches for simpler analytic forms. Across the dynamical-system experiments, SORT matches or improves upon library-based sparse-regression baselines when the basis is well adapted to the problem, and shows more stable degradation under sparse sampling, noisy derivative estimates, and representation mismatch. Specific examples illustrate why this representation is useful: if a finite library misses the problem-specific nonlinearity, the resulting model can fail. SORT is not immune to mismatch, but it shifts the problem away from brittle selection among generic terms to basis design adapted to the problem domain. The experiments also show that dominant low-order coefficients persist as model order increases, supporting order-consistent model growth. Beyond equation discovery, the same learned expansion supports nonlinear approximation and estimation of complex, high-dimensional integrals by coefficient readout. Overall, SORT provides a reusable intermediate representation for system identification, approximation, and integration, while making basis design an explicit part of the scientific modeling problem.