Provable Adaptivity of Adam under Non-uniform Smoothness-Reference-Cited by-同舟云学术

Provable Adaptivity of Adam under Non-uniform Smoothness

Published:2024-08-24 Issue: Volume:8 Page:2960-2969
ISSN:
Container-title:Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
language:
Short-container-title:

Author:

Wang Bohan¹^ORCID,Zhang Yushun²^ORCID,Zhang Huishuai³^ORCID,Meng Qi⁴^ORCID,Sun Ruoyu⁵^ORCID,Ma Zhi-Ming⁴^ORCID,Liu Tie-Yan⁶^ORCID,Luo Zhi-Quan⁵^ORCID,Chen Wei⁷^ORCID

Affiliation:

1. University of Science and Technology of China & Microsoft Research Asia, Beijing, Haidian, China

2. The Chinese University of Hong Kong, Shenzhen, Shenzhen, Guangdong, China

3. Peking University, Beijing, China

4. Chinese Academy of Mathematics and Systems Science, Beijing, China

5. The Chinese University of Hong Kong, Shenzhen, Shenzhen, China

6. Microsoft, Beijing, China

7. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Funder

NSFC

CAS Project for Young Scientists in Basic Research

Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project

the Strategic Priority Research Program of the Chinese Academy of Sciences

Innovation Project of ICT CAS

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3637528.3671718

Reference49 articles.

1. Léon Bottou. 2009. Curiously fast convergence of some stochastic gradient descent algorithms. In Proceedings of the symposium on learning and data science, Paris, Vol. 8. 2624--2633.

2. Léon Bottou. 2012. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade. Springer, 421--436.

3. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations.

4. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901.

5. Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. 2021. Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration. arXiv preprint arXiv:2101.05471 (2021).