Efficient Federated Learning Using Layer-Wise Regulation and Momentum Aggregation

Abstract

Federated Learning (FL) enables multiple parties to train a global model collaboratively without sharing local data. However, a key challenge of FL is data distribution heterogeneity across participants, which causes model drift in local training and significantly reduces the model performance. To address this challenge, we analyze the inconsistency differences between different model layers of local models and further propose Layer-wise Distance Regularization (LWDR) and Layer-wise Momentum Aggregation (LWMA). The proposed LWDR and LWMA optimize the local training and model aggregation processes, respectively, to improve the convergence performance of FL on data in the nonindependent and identically distributed (Non-IID) scenarios. Our experiments on well-known datasets show that our algorithm significantly outperforms the state-of-the-art FL algorithms in convergence speed, accuracy, and stability in different Non-IID scenarios.