Affiliation:
1. State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang 550025, China
2. Guizhou Key Laboratory of Pattern Recognition and Intelligent System, Guizhou Minzu University, Guiyang 550025, China
3. Guizhou Big Data Academy, Guizhou University, Guiyang 550025, China
4. Institute of Guizhou Aerospace Measuring and Testing Technology, Guiyang 550025, China
Abstract
Deep learning is one of the most exciting and promising techniques in the field of artificial intelligence (AI), which drives AI applications to be more intelligent and comprehensive. However, existing deep learning techniques usually require a large amount of expensive labeled data, which limit the application and development of deep learning techniques, and thus it is imperative to study unsupervised machine learning. The learning of deep representations by mutual information estimation and maximization (Deep InfoMax or DIM) method has achieved unprecedented results in the field of unsupervised learning. However, in the DIM method, to restrict the encoder to learn more normalized feature representations, an adversarial network learning method is used to make the encoder output consistent with a priori positively distributed data. As we know, the model training of the adversarial network learning method is difficult to converge, because there is a logarithmic function in the loss function of the cross-entropy measure, and the gradient of the model parameters is susceptible to the “gradient explosion” or “gradient disappearance” phenomena, which makes the training of the DIM method extremely unstable. In this regard, we propose a Wasserstein distance-based DIM method to solve the stability problem of model training, and our method is called the WDIM. Subsequently, the training stability of the WDIM method and the classification ability of unsupervised learning are verified on the CIFAR10, CIFAR100, and STL10 datasets. The experiments show that our proposed WDIM method is more stable to parameter updates, has faster model convergence, and at the same time, has almost the same accuracy as the DIM method on the classification task of unsupervised learning. Finally, we also propose a reflection of future research for the WDIM method, aiming to provide a research idea and direction for solving the image classification task with unsupervised learning.
Funder
National Key Research and Development Program of China
National Natural Science Foundation of China
Research Project of Guizhou University for Talent Introduction
Cultivation Project of Guizhou University
Open Fund of Key Laboratory of Advanced Manufacturing Technology, Ministry of Education
Subject
General Physics and Astronomy
Reference38 articles.
1. A fast, on-line algorithm for PCA and its convergence characteristics;Rao;Neural Networks for Signal Processing X, Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No. 00TH8501), Sydney, NSW, Australia, 11–13 December 2000,2000
2. A global geometric framework for nonlinear dimensionality reduction;Tenenbaum;Science,2000
3. Nonlinear dimensionality reduction by locally linear embedding;Roweis;Science,2000
4. Laplacian eigenmaps for dimensionality reduction and data representation;Belkin;Neural Comput.,2003
5. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data;Donoho;Proc. Natl. Acad. Sci. USA,2003