Complexity control by gradient descent in deep networks-Reference-Cited by-同舟云学术

Complexity control by gradient descent in deep networks

Published:2020-02-24 Issue:1 Volume:11 Page:
ISSN:2041-1723
Container-title:Nature Communications
language:en
Short-container-title:Nat Commun

Author:

Poggio Tomaso,Liao Qianli,Banburski Andrzej

Abstract

AbstractOverparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.

Funder

NSF | NSF Office of the Director | Office of International Science and Engineering

JUMP Semiconductor Research Corporation (SRC) program sponsored by DARPA

Publisher

Springer Science and Business Media LLC

Subject

General Physics and Astronomy,General Biochemistry, Genetics and Molecular Biology,General Chemistry

Link

http://www.nature.com/articles/s41467-020-14663-9.pdf

Reference14 articles.

1. Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR) (Toulon, France, 2017).

2. Rosasco, L. & Villa, S. Learning with incremental iterative regularization. In Advances in Neural Information Processing Systems, 1630–1638 (2015).

3. Engl, H., Hanke, M. & Neubauer, A. Regularization of Inverse Problems (Kluwer Academic Publishers, 2000).

4. Douglas, S. C., Amari, S. & Kung, S. Y. On gradient adaptation with unit-norm constraints. IEEE Transact. Signal Process. 48(June), 1843–1847 (2000).

5. Banburski, A. et al. Theory of deep learning III: dynamics and generalization in deep networks. CBMM Memo No. 090 (2019).

Cited by 18 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Structural health monitoring of railway truss bridge under moving train load using decision tree models and residual neural networks;2024-08-22

2. Network properties determine neural network performance;Nature Communications;2024-07-08

3. Robust implicit regularization via weight normalization;Information and Inference: A Journal of the IMA;2024-07-01

4. Towards a Deeper Understanding of Global Covariance Pooling in Deep Learning: An Optimization Perspective;IEEE Transactions on Pattern Analysis and Machine Intelligence;2023-12

5. Learning Canonical Embeddings for Unsupervised Shape Correspondence With Locally Linear Transformations;IEEE Transactions on Pattern Analysis and Machine Intelligence;2023-12