Author:
Angelini Maria Chiara,Cavaliere Angelo Giorgio,Marino Raffaele,Ricci-Tersenghi Federico
Abstract
AbstractIs Stochastic Gradient Descent (SGD) substantially different from Metropolis Monte Carlo dynamics? This is a fundamental question at the time of understanding the most used training algorithm in the field of Machine Learning, but it received no answer until now. Here we show that in discrete optimization and inference problems, the dynamics of an SGD-like algorithm resemble very closely that of Metropolis Monte Carlo with a properly chosen temperature, which depends on the mini-batch size. This quantitative matching holds both at equilibrium and in the out-of-equilibrium regime, despite the two algorithms having fundamental differences (e.g. SGD does not satisfy detailed balance). Such equivalence allows us to use results about performances and limits of Monte Carlo algorithms to optimize the mini-batch size in the SGD-like algorithm and make it efficient at recovering the signal in hard inference problems.
Funder
PNRR MUR
European Union – NextGenerationEU
PRIN 2022 PNRR
Publisher
Springer Science and Business Media LLC
Reference65 articles.
1. Cormen, T. H., Leiserson, C. E., Rivest, R. L. & Stein, C. Introduction to Algorithms (MIT press, 2022).
2. Cugliandolo, L. F. A scientific portrait of Giorgio Parisi: Complex systems and much more. J. Phys.: Complex. 4, 011001 (2023).
3. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. & Teller, E. Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953).
4. Amari, S.-I. Backpropagation and stochastic gradient descent method. Neurocomputing 5, 185–196 (1993).
5. Bottou, L. Stochastic Gradient Descent Tricks. Neural Networks: Tricks of the Trade 2nd edn, 421–436 (Springer, 2012).