A Simple Approximation to the Distribution of the Ridge Regression Estimator
2026-08-03 • Machine Learning
Machine Learning
AI summaryⓘ
The authors provide a new way to understand how ridge regression behaves when you only have a limited amount of data. They use a simple Gaussian model to describe the estimator’s behavior, especially focusing on how the balance between bias and variance affects errors. Their method works even when the data has complicated patterns like changing variance or dependence over time, although it applies to models with a fixed number of variables. Using this approach, they suggest two new methods to choose how much to shrink the coefficients in ridge regression to improve prediction accuracy.
ridge regressionbias-variance tradeoffGaussian approximationregularization parameterfinite-sample distributionheteroskedasticityautocorrelationasymptoticsprediction risk
Authors
José Luis Montiel Olea, Ryan Strong, Amilcar Velez, Zhuoheng Xu, Haomin Yu
Abstract
We present a simple Gaussian approximation to the finite-sample distribution of the classical ridge regression estimator. Our approximation captures the fact that, in finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error. Our approximation is based on nonstandard asymptotics where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size; and $ii)$ we treat the population regression coefficients as \emph{local} to the reference vector that defines the estimator's direction of shrinkage. In contrast to other asymptotic approximations in the literature, we allow for general forms of heteroskedasticity and autocorrelation in the data generating process (at the cost of considering a low-dimensional model where the number of covariates is not allowed to grow with the sample size). We use our simple Gaussian approximation to propose two new strategies to select the regularization parameter for the ridge regression estimator. The suggested strategies select the regularization parameter to minimize either average or worst-case excess prediction risk, where risk is computed using our suggested Gaussian approximation.