Comparative study of optimization techniques in deep learning: Application in the ophthalmology field

Abstract

Abstract The optimization is a discipline which is part of mathematics and which aims to model, analyse and solve analytically or numerically problems of minimization or maximization of a function on a specific dataset. Several optimization algorithms are used in systems based on deep learning (DL) such as gradient descent (GD) algorithm. Considering the importance and the efficiency of the GD algorithm, several research works made it possible to improve it and to produce several other variants which also knew great success in DL. This paper presents a comparative study of stochastic, momentum, Nesterov, AdaGrad, RMSProp, AdaDelta, Adam, AdaMax and Nadam gradient descent algorithms based on the speed of convergence of these different algorithms, as well as the mean absolute error of each algorithm in the generation of an optimization solution. The obtained results show that AdaGrad algorithm represents the best performances than the other algorithms with a mean absolute error (MAE) of 0.3858 in 53 iterations and AdaDelta one represents the lowest performances with a MAE of 0.6035 in 6000 iterations. The case study treated in this work is based on an extract of data from the keratoconus dataset of Harvard Dataverse and the results are obtained using Python.