The Effect of the Second Stage Estimator on Model Performance in Post-LASSO Method

Abstract

Penalized linear regression methods are used for the accurate prediction of new observations and to obtain interpretable models. The performance of these methods depends on the properties of the true coefficient vector. The LASSO method is a penalized regression method that can simultaneously perform coefficient shrinkage and variable selection in a continuous process. Depending on the structure of the dataset, different estimators have been proposed to overcome the problems faced by LASSO. The estimation method used in the second stage of the post-LASSO two-stage regression method proposed as an alternative to LASSO has a considerable effect on model performance. In this study, the performance of the post-LASSO is compared with classical penalized regression methods ridge, LASSO, elastic net, adaptive LASSO and Post-LASSO by using different estimation methods in the second stage of the post-LASSO. In addition, the effect of the magnitude and position of the signal values in the real coefficient vector on the performance of the models obtained by these methods is analyzed. The mean squared error and standard deviation of the predictions calculated on the test set are used to compare the prediction performance of the models, while the active set sizes are used to compare their performance in variable selection. According to the findings obtained from the simulation studies, the choice of the second-stage estimator and the structure of the true coefficient vector significantly affect the success of the post-LASSO method compared to other methods.