System response curve based first‐order optimization algorithms for cyber‐physical‐social intelligence

Abstract

AbstractThe continuous enhancement of optimization algorithms and their parameters has spurred the expansion of AI into novel application domains such as image recognition and smart home technology. This paper employs the system response curve (SRC) to the adaptive learning rate optimizer, addressing challenges associated with the establishment of the optimizer control model and parameter adjustments affecting the dynamic performance of the system. These insights offer theoretical support for the optimizer's application in deep learning models. To begin, the adaptive learning rate optimizer is a time‐varying system. Based on the intrinsic relationship between the network optimization and the control system, the time domain expression and approximate transfer function of the adaptive learning rate optimizer are derived, and the system dynamic model is established. Furthermore, based on the system control model of the optimizer, it is proposed to explain the performance impacts of different optimizers and their hyperparameters on the deep learning model through the SRC. Finally, experiments are performed on the MNIST, CIFAR‐10, UTKinect‐Action3D, and Florence3D‐Action datasets to validate the control theory of explaining optimizers through system response curves. The experimental results show that the recognition performance of the Adaptive Moment Estimate (Adam) is better than that of the Adaptive Gradient (AdaGrad) and Root Mean Square Propagation (RMSprop). Additionally, the learning rate affects the model training speed, and the practical application aligns with the theoretical analysis.