On Stage-Wise Backpropagation for Improving Cheng’s Method for Fully Connected Cascade Networks

Abstract

AbstractIn this journal, Cheng has proposed a backpropagation (BP) procedure called BPFCC for deep fully connected cascaded (FCC) neural network learning in comparison with a neuron-by-neuron (NBN) algorithm of Wilamowski and Yu. Both BPFCC and NBN are designed to implement the Levenberg-Marquardt method, which requires an efficient evaluation of the Gauss-Newton (approximate Hessian) matrix $$\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}$$ ∇ r T ∇ r , the cross product of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r of the residual vector $$\textbf{r}$$ r in nonlinear least squares sense. Here, the dominant cost is to form $$\nabla \textbf{r}^\textsf{T} \nabla \textbf{r}$$ ∇ r T ∇ r by rank updates on each data pattern. Notably, NBN is better than BPFCC for the multiple $$q~\!(>\!1)$$ q ( > 1 ) -output FCC-learning when q rows (per pattern) of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r are evaluated; however, the dominant cost (for rank updates) is common to both BPFCC and NBN. The purpose of this paper is to present a new more efficient stage-wise BP procedure (for q-output FCC-learning) that reduces the dominant cost with no rows of $$\nabla \textbf{r}$$ ∇ r explicitly evaluated, just as standard BP evaluates the gradient vector $$\nabla \textbf{r}^\textsf{T} \textbf{r}$$ ∇ r T r with no explicit evaluation of any rows of the Jacobian matrix $$\nabla \textbf{r}$$ ∇ r .