Abstract
The high accuracy attainment, using less complex architectures of neural networks, remains one of the most important problems in machine learning. In many studies, increasing the quality of recognition and prediction is obtained by extending neural networks with usual or special neurons, which significantly increases the time of training. However, engaging an optimization algorithm, which gives us a value of the loss function in the neighborhood of global minimum, can reduce the number of layers and epochs. In this work, we explore the extreme searching of multidimensional functions by proposed natural gradient descent based on Dirichlet and generalized Dirichlet distributions. The natural gradient is based on describing a multidimensional surface with probability distributions, which allows us to reduce the change in the accuracy of gradient and step size. The proposed algorithm is equipped with step-size adaptation, which allows it to obtain higher accuracy, taking a small number of iterations in the process of minimization, compared with the usual gradient descent and adaptive moment estimate. We provide experiments on test functions in four- and three-dimensional spaces, where natural gradient descent proves its ability to converge in the neighborhood of global minimum. Such an approach can find its application in minimizing the loss function in various types of neural networks, such as convolution, recurrent, spiking and quantum networks.
Funder
Russian Science Foundation
Subject
General Mathematics,Engineering (miscellaneous),Computer Science (miscellaneous)
Reference24 articles.
1. AdaGrad Stepsizes: Sharp Convergence Over Nonconvex Landscapes;Ward;J. Mach. Learn. Res.,2020
2. Adaptive subgradient methods for online learning and stochastic optimization;Duchi;J. Mach. Learn. Res.,2011
3. Convergence of the RMSProp deep learning method with penalty for nonconvex optimization
4. Genetic Optimization Method of Pantograph and Catenary Comprehensive Monitor Status Prediction Model Based on Adadelta Deep Neural Network
5. The BP Neural Network with Adam Optimizer for Predicting Audit Opinions of Listed Companies;Wu;IAENG Int. J. Comput. Sci.,2021
Cited by
4 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献