Decentralized and parallel primal and dual accelerated methods for stochastic convex programming problems-Reference-Cited by-同舟云学术

Decentralized and parallel primal and dual accelerated methods for stochastic convex programming problems

Published:2021-01-22 Issue:3 Volume:29 Page:385-405
ISSN:1569-3945
Container-title:Journal of Inverse and Ill-posed Problems
language:en
Short-container-title:

Author:

Dvinskikh Darina¹,Gasnikov Alexander²

Affiliation:

1. Weierstrass Institute for Applied Analysis and Stochastics , Mohrenstr. 39, 10117 Berlin , Germany ; and Moscow Institute of Physics and Technology, Dolgoprudny, Moscow Region, Russia; and Institute for Information Transmission Problems RAS, Moscow, Russia

2. Department of Control and Applied Mathematics , Moscow Institute of Physics and Technology , 9 Institutskiy pereulok , Dolgoprudny , Moscow Region, 141701 , Russia ; and Institute for Information Transmission Problems RAS, Moscow, Russia; and Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany

Abstract

Abstract We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems. Both for primal and dual oracles, the proposed methods are optimal in terms of the number of communication steps. However, for all classes of the objective, the optimality in terms of the number of oracle calls per node takes place only up to a logarithmic factor and the notion of smoothness. By using mini-batching technique, we show that the proposed methods with stochastic oracle can be additionally parallelized at each node. The considered algorithms can be applied to many data science problems and inverse problems.

Funder

Russian Science Foundation

Russian Foundation for Basic Research

Ministry of Science and Higher Education of the Russian Federation

Publisher

Walter de Gruyter GmbH

Subject

Applied Mathematics

Link

https://www.degruyter.com/document/doi/10.1515/jiip-2020-0068/pdf

Reference106 articles.

1. Z. Allen-Zhu, Katyusha: The first direct acceleration of stochastic gradient methods, STOC’17—Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, ACM, New York (2017), 1200–1205.

2. Z. Allen-Zhu, How to make the gradients small stochastically: Even faster convex and nonconvex SGD, Advances in Neural Information Processing Systems 31 (NeurIPS 2018), Neural Information Processing Systems Foundation, San Diego (2018), 1157–1167.

3. Z. Allen-Zhu and E. Hazan, Optimal black-box reductions between optimization objectives, Advances in Neural Information Processing Systems 29 (NeurIPS 2016), Neural Information Processing Systems Foundation, San Diego (2016), 1614–1622.

4. A. S. Anikin, A. V. Gasnikov, P. E. Dvurechensky, A. I. Tyurin and A. V. Chernov, Dual approaches to the minimization of strongly convex functionals with a simple structure under affine constraints, Comput. Math. Math. Phys. 57 (2017), no. 8, 1262–1276.

5. Y. Arjevani and O. Shamir, Communication complexity of distributed convex learning and optimization, Advances in Neural Information Processing Systems 28 (NeurIPS 2015), Neural Information Processing Systems Foundation, San Diego (2015), 1756–1764.

Cited by 17 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Provably Accelerated Decentralized Gradient Methods Over Unbalanced Directed Graphs;SIAM Journal on Optimization;2024-03-22

2. Decentralized saddle point problems via non-Euclidean mirror prox;Optimization Methods and Software;2024-01-24

3. Inexact tensor methods and their application to stochastic convex optimization;Optimization Methods and Software;2023-11-17

4. Optimal gradient tracking for decentralized optimization;Mathematical Programming;2023-07-07

5. Accelerated gradient methods with absolute and relative noise in the gradient;Optimization Methods and Software;2023-06-08