Multi-index antithetic stochastic gradient algorithm
-
Published:2023-03-03
Issue:2
Volume:33
Page:
-
ISSN:0960-3174
-
Container-title:Statistics and Computing
-
language:en
-
Short-container-title:Stat Comput
Author:
Majka Mateusz B.ORCID, Sabate-Vidales Marc, Szpruch Łukasz
Abstract
AbstractStochastic Gradient Algorithms (SGAs) are ubiquitous in computational statistics, machine learning and optimisation. Recent years have brought an influx of interest in SGAs, and the non-asymptotic analysis of their bias is by now well-developed. However, relatively little is known about the optimal choice of the random approximation (e.g mini-batching) of the gradient in SGAs as this relies on the analysis of the variance and is problem specific. While there have been numerous attempts to reduce the variance of SGAs, these typically exploit a particular structure of the sampled distribution by requiring a priori knowledge of its density’s mode. In this paper, we construct a Multi-index Antithetic Stochastic Gradient Algorithm (MASGA) whose implementation is independent of the structure of the target measure. Our rigorous theoretical analysis demonstrates that for log-concave targets, MASGA achieves performance on par with Monte Carlo estimators that have access to unbiased samples from the distribution of interest. In other words, MASGA is an optimal estimator from the mean square error-computational cost perspective within the class of Monte Carlo estimators. To illustrate the robustness of our approach, we implement MASGA also in some simple non-log-concave numerical examples, however, without providing theoretical guarantees on our algorithm’s performance in such settings.
Publisher
Springer Science and Business Media LLC
Subject
Computational Theory and Mathematics,Statistics, Probability and Uncertainty,Statistics and Probability,Theoretical Computer Science
Reference61 articles.
1. Aicher, C., Ma, Y.-A., Foti, N.J., Fox, E.B.: Stochastic gradient MCMC for state space models. SIAM J. Math. Data Sci. 1(3), 555–587 (2019) 2. Baker, J., Fearnhead, P., Fox, E.B., Nemeth, C.: Control variates for stochastic gradient MCMC. Stat. Comput. 29(3), 599–615 (2019) 3. Barkhagen, M., Chau, N.H., Moulines, É., Rásonyi, M., Sabanis, S., Zhang, Y.: On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case. Bernoulli (2020) 4. Ben Alaya, M., Kebaier, A.: Central limit theorem for the multilevel Monte Carlo Euler method. Ann. Appl. Probab. 25(1), 211–234 (2015) 5. Ben Alaya, M., Kebaier, A., Tram Ngo, T.B.: Central Limit Theorem for the $$\sigma $$-antithetic multilevel Monte Carlo method. arXiv e-prints, page arXiv:2002.08834 (2020)
|
|