Ridge Regression and the Elastic Net: How Do They Do as Finders of True Regressors and Their Coefficients?

Abstract

For the linear model Y=Xb+error, where the number of regressors (p) exceeds the number of observations (n), the Elastic Net (EN) was proposed, in 2005, to estimate b. The EN uses both the Lasso, proposed in 1996, and ordinary Ridge Regression (RR), proposed in 1970, to estimate b. However, when p>n, using only RR to estimate b has not been considered in the literature thus far. Because RR is based on the least-squares framework, only using RR to estimate b is computationally much simpler than using the EN. We propose a generalized ridge regression (GRR) algorithm, a superior alternative to the EN, for estimating b as follows: partition X from left to right so that every partition, but the last one, has 3 observations per regressor; for each partition, we estimate Y with the regressors in that partition using ordinary RR; retain the regressors with statistically significant t-ratios and the corresponding RR tuning parameter k, by partition; use the retained regressors and k values to re-estimate Y by GRR across all partitions, which yields b. Algorithmic efficacy is compared using 4 metrics by simulation, because the algorithm is mathematically intractable. Three metrics, with their probabilities of RR’s superiority over EN in parentheses, are: the proportion of true regressors discovered (99%); the squared distance, from the true coefficients, of the significant coefficients (86%); and the squared distance, from the true coefficients, of estimated coefficients that are both significant and true (74%). The fourth metric is the probability that none of the regressors discovered are true, which for RR and EN is 4% and 25%, respectively. This indicates the additional advantage RR has over the EN in terms of discovering causal regressors.