Asymptotic Convergence Rate of Dropout on Shallow Linear Neural Networks-Reference-Cited by-同舟云学术

Asymptotic Convergence Rate of Dropout on Shallow Linear Neural Networks

Published:2022-05-26 Issue:2 Volume:6 Page:1-53
ISSN:2476-1249
Container-title:Proceedings of the ACM on Measurement and Analysis of Computing Systems
language:en
Short-container-title:Proc. ACM Meas. Anal. Comput. Syst.

Author:

Senen-Cerda Albert¹,Sanders Jaron¹

Affiliation:

1. Eindhoven University of Technology, Eindhoven, Netherlands

Abstract

We analyze the convergence rate of gradient flows on objective functions induced by Dropout and Dropconnect, when applying them to shallow linear Neural Networks(NN) ---which can also be viewed as doing matrix factorization using a particular regularizer. Dropout algorithms such as these are thus regularization techniques that use 0,1-valued random variables to filter weights during training in order to avoid coadaptation of features. By leveraging a recent result on nonconvex optimization and conducting a careful analysis of the set of minimizers as well as the Hessian of the loss function, we are able to obtain (i) a local convergence proof of the gradient flow and (ii) a bound on the convergence rate that depends on the data, the dropout probability, and the width of the NN. Finally, we compare this theoretical bound to numerical simulations, which are in qualitative agreement with the convergence bound and match it when starting sufficiently close to a minimizer.

Publisher

Association for Computing Machinery (ACM)

Subject

Computer Networks and Communications,Hardware and Architecture,Safety, Risk, Reliability and Quality,Computer Science (miscellaneous)

Link

https://dl.acm.org/doi/pdf/10.1145/3530898

Reference45 articles.

1. Majorization, doubly stochastic matrices, and comparison of eigenvalues

2. Sanjeev Arora , Noah Golowich , Nadav Cohen , and Wei Hu . 2019 . A convergence analysis of gradient descent for deep linear neural networks . In 7th International Conference on Learning Representations, ICLR 2019. Sanjeev Arora, Noah Golowich, Nadav Cohen, and Wei Hu. 2019. A convergence analysis of gradient descent for deep linear neural networks. In 7th International Conference on Learning Representations, ICLR 2019.

3. Andreas Arvanitogeorgos . 2003. ¯ An introduction to Lie groups and the geometry of homogeneous spaces . Vol. 22 . American Mathematical Soc . Andreas Arvanitogeorgos. 2003. ¯ An introduction to Lie groups and the geometry of homogeneous spaces. Vol. 22. American Mathematical Soc.

4. Jimmy Ba and Brendan Frey. 2013. Adaptive Dropout for training deep neural networks. In Advances in Neural Information Processing Systems. 3084--3092. Jimmy Ba and Brendan Frey. 2013. Adaptive Dropout for training deep neural networks. In Advances in Neural Information Processing Systems. 3084--3092.

5. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers