Affiliation:
1. HASAN KALYONCU UNIVERSITY
Abstract
This work investigates the integration of multiplicative calculus into gradient descent algorithms, including Adaptive Gradient algorithm (AdaGrad), Root Mean Squared Propagation (RMSProp), Nesterov Accelerated Gradient (NAG), and Momentum, to optimize exponential-quadratic-logarithmic composite functions with the positivity constrained. This research, conducted across five scenarios within the Constrained and Unconstrained Testing Environment (CUTEst), compares these multiplicative methods with their classical counterparts under a variety of constraints environments such as bounded, quadratic, and other types, and unconstrained environments. The results demonstrate the significant superiority of multiplicative-based algorithms, especially in unconstrained and bounded constrained scenarios, and demonstrate their potential for complex optimization tasks. Statistical analysis supports the observed performance advantages, indicating significant opportunities for optimization strate-gies in positive domains.
Publisher
International Journal of Pure and Applied Sciences
Reference22 articles.
1. Ruder, S. (2016). An overview of gradient descent optimization algorithms. 1–14, [Online]. Available: http://arxiv.org/abs/1609.04747.
2. Baldi, P. (1995). Gradient Descent Learning Algorithm Overview: A General Dynamical Systems Perspective. IEEE Trans. Neural Networks, 6(1), 182–195.
3. Zou, F., Shen, L., Jie, Z., Zhang, W. and Liu, W. (2019). A sufficient condition for convergences of adam and rmsprop. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2019(1), 11119–11127.
4. Wilson, A. C., Roelofs, R., Stern, M., Srebro, N. and Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning. Adv. Neural Inf. Process. Syst., 2017, 4149–4159.
5. Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1), 145–151.