Extract the information from big data with randomly distributed noise

Abstract

Abstract In this manuscript, a purely data-driven statistical regularization method is proposed for extracting the information from big data with randomly distributed noise. Since the variance of the noise may be large, the method can be regarded as a general data preprocessing method in ill-posed problems, which is able to overcome the difficulty that the traditional regularization method is unable to solve, and has superior advantage in computing efficiency. The unique solvability of the method is proved, and a number of conditions are given to characterize the solution. The regularization parameter strategy is discussed, and the rigorous upper bound estimation of the confidence interval of the error in the L 2 L^{2} norm is established. Some numerical examples are provided to illustrate the appropriateness and effectiveness of the method.