Enhanced Component-Wise Natural Gradient Descent Training Method for Deep Neural Networks

Abstract

Abstract This research significantly advances Component-Wise Natural Gradient Descent (CW-NGD), a network training method that facilitates efficient parameter updates by approximating the curvature Fisher Information Matrix. By the investigation of the exponential moving average integration, and appropriate hyperparameters selection obtained from the comprehensive analysis results, significant enhancements in CW-NGD's performance have been achieved. Particularly we enhance CW-NGD to operate across multiple GPUs, bypassing the memory constraints when working with large-scale models. These improvements enable CW-NGD to attain state-of-the-art accuracy on deep networks, which prior work could not achieve In an expansive comparison across four diverse datasets and models, CW-NGD achieves similar or superior accuracy while outperforming all other established network training methods, encompassing Adam, Stochastic Gradient Descent, and Kronecker-factored Approximate Curvature, in terms of convergence speed and stability. This study establishes CW-NGD as a robust and versatile network training technique, showcasing its adaptability and potential applications across various domains.