Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons-Reference-Cited by-同舟云学术

Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons

Published:1998-11-01 Issue:8 Volume:10 Page:2137-2157
ISSN:0899-7667
Container-title:Neural Computation
language:en
Short-container-title:Neural Computation

Author:

Yang Howard Hua¹,Amari Shun-ichi²

Affiliation:

1. Department of Computer Science, Oregon Graduate Institute, Portland, OR 97291, U.S.A.

2. Laboratory for Information Synthesis, RIKEN Brain Science Institute, Saitama 351-01, Japan

Abstract

The natural gradient descent method is applied to train an n-m-1 multilayer perceptron. Based on an efficient scheme to represent the Fisher information matrix for an n-m-1 stochastic multilayer perceptron, a new algorithm is proposed to calculate the natural gradient without inverting the Fisher information matrix explicitly. When the input dimension n is much larger than the number of hidden neurons m, the time complexity of computing the natural gradient is O(n).

Publisher

MIT Press - Journals

Subject

Cognitive Neuroscience,Arts and Humanities (miscellaneous)

Link

https://www.mitpressjournals.org/doi/pdf/10.1162/089976698300017007

Reference3 articles.

1. Natural Gradient Works Efficiently in Learning

2. On-line learning in soft committee machines

3. Statistical inference: learning in artificial neural networks

Cited by 47 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Cognitive process-driven model design: A deep learning recommendation model with textual review and context;Decision Support Systems;2024-01

2. A Regularized Non-Parametric Denoising Method;2023 China Automation Congress (CAC);2023-11-17

3. Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems;SIAM Journal on Scientific Computing;2023-07-10

4. State space representation and phase analysis of gradient descent optimizers;Science China Information Sciences;2023-03-27

5. IoT data analytics in dynamic environments: From an automated machine learning perspective;Engineering Applications of Artificial Intelligence;2022-11