A Derivative-Incorporated Adaptive Gradient Method for Federated Learning
-
Published:2023-08-04
Issue:15
Volume:11
Page:3403
-
ISSN:2227-7390
-
Container-title:Mathematics
-
language:en
-
Short-container-title:Mathematics
Author:
Gao Huimin1, Wu Qingtao1ORCID, Cao Hongyan2, Zhao Xuhui1, Zhu Junlong1, Zhang Mingchuan1ORCID
Affiliation:
1. College of Information Engineering, Henan University of Science and Technology, Luoyang 471023, China 2. China Research Institute of Radiowave Propagation, Qingdao 266107, China
Abstract
As a new machine learning technique, federated learning has received more attention in recent years, which enables decentralized model training across data silos or edge intelligent devices in the Internet of Things without exchanging local raw data. All kinds of algorithms are proposed to solve the challenges in federated learning. However, most of these methods are based on stochastic gradient descent, which undergoes slow convergence and unstable performance during the training stage. In this paper, we propose a differential adaptive federated optimization method, which incorporates an adaptive learning rate and the gradient difference into the iteration rule of the global model. We further adopt the first-order moment estimation to compute the approximate value of the differential term so as to avoid amplifying the random noise from the input data sample. The theoretical convergence guarantee is established for our proposed method in a stochastic non-convex setting under full client participation and partial client participation cases. Experiments for the image classification task are performed on two standard datasets by training a neural network model, and experiment results on different baselines demonstrate the effectiveness of our proposed method.
Funder
National Natural Science Foundation of China Leading talents of science and technology in the Central Plain of China Science & Technology Innovation Talents in the University of Henan Province of China basic research projects in the University of Henan Province, China International Cooperation Project of Henan Province
Subject
General Mathematics,Engineering (miscellaneous),Computer Science (miscellaneous)
Reference39 articles.
1. Portelli, K., and Anagnostopoulos, C. (2017, January 21–23). Leveraging Edge Computing through Collaborative Machine Learning. Proceedings of the 5th International Conference on Future Internet of Things and Cloud Workshops, FiCloud Workshops, Prague, Czech Republic. 2. Hu, Y., Niu, D., Yang, J., and Zhou, S. (2019, January 4–8). FDML: A Collaborative Machine Learning Framework for Distributed Features. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA. 3. Konečný, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., and Bacon, D. (2016). Federated Learning: Strategies for Improving Communication Efficiency. arXiv. 4. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Lauderdale, FL, USA. 5. Pinto Neto, E.C., Sadeghi, S., Zhang, X., and Dadkhah, S. (2023). Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges. Appl. Sci., 13.
|
|