Improve Generalization in Deep Neural Network using Multivariate Statistical Knowledge Transformation

Abstract

Abstract Most DNNs are trained in an over-parametrized regime. In this case, the numbers of their parameters are more than available training data which reduces the generalization capability and performance on new and unseen samples. generalization of deep neural networks (DNNs) has been improved through applying various methods such as regularization techniques, data enhancement, network capacity restriction, injection randomness, etc. In this paper, we proposed an effective generalization method, named multivariate statistical knowledge transformation, which learns feature distribution to separate samples based on variance of deep hypothesis space in all dimensions. Moreover, the proposed method uses latent knowledge of the target to boost the confidence of its prediction. Our method was evaluated on CIFAR-10 and CIFAR-100 datasets. The multivariate statistical knowledge transformation produces competitive results, compared with the state-of-the-art methods. Experimental results show that the proposed method improves the generalization of a DNN by 5% in test error and makes it much faster to converge in total runs.