Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter

Abstract

Variational estimation method is a deterministic approximation technique which involves Bayesian framework while giving a point estimate instead of the usual Bayesian interval estimation. The linear regression model, which has always been a popular model, can benefit from the implementation of variational estimation method. In this paper, the theoretical basis on why variational method can reduce overfitting in linear regression is reviewed. Based on the review, in theory, variational method is more robust to overfitting than MLE. This paper also performed a simulation study. The simulation is done in a manner such that the simulation represents the situation of predicting for new or hidden data. The simulation starts from generating random explanatory data and generates the appropriate response data based on linear regression equation. Then, the randomly generated data is used to estimate the linear regression parameters. The simulation is performed to compare the parameters estimation results from variational method with the method of MLE. The comparison is done using the estimation values and the squared differences between true parameters value and the estimates. Empirical findings show that both methods have relatively close estimate values. It can be seen as the simulation study concludes that both variational and ML yield rather close parameters estimates for simple linear regression case. The estimates closeness gets more obvious as the sample size grows. The study also found that Variational method has performs better in terms of parameters estimation in linear regression when the sample size is small or the data has large variance.