Application of Support Vector Regression in Krylov Solvers

Abstract

Support vector regression (SVR) is well known as a regression or prediction tool under the Machine Learning (ML) which preserves all the key features through the training data. Different from general prediction, here, we proposed SVR to predict the new approximate solutions after we generated some iterates using an iterative method called Lanczos algorithm, one class of Krylov solvers. As we know that all Krylov solvers, including Lanczos methods, for solving the high dimensions of systems of linear equations (SLEs) problems experiences breakdown which causes the sequence of the iterates is incomplete, or the good approximate solution is never reached. By assuming that some iterates exist after the breakdown, then we could predict what they are. It is realized by learning the previous iterates generated by the Lanczos solvers, which is also called the training data. The SVR is then used to predict the next iterate which is expected the sequence now has similar property as the previous one before breaking down. Furthermore, we implemented the hybrid SVR-Lanczos (or SVR-L) in the restarting frame work, then it is called as hybrid restarting-SVR-L. The idea behind the restarting is that one time running hybrid SVR-L cannot obtain a good approximate solution with small residual norm. By taking one iterate which is resulted by the hybrid SVR-L, putting it as the initial guess, will give us the better solution. To test our idea of prediction of SLEs solutions, we also used the regular regression and compared with the SVR. Numerical results are presented and compared between these two predictors. Lastly, we compared our proposed method with existing interpolation and extrapolation methods to predict the approximate solution of SLEs. The results showed that our restarting SVR-L performed better compared with the regular regression.