Combined algorithm for training neural networks of direct propagation

Abstract

The essence of learning neural networks of direct propagation is to minimize the function of the root mean square error of the output. This function is multimodal, ie it has several local minima. To find the minimum of such functions, gradient and stochastic methods are most often used, which do not guarantee finding the global minimum. The article analyzes the gradient algorithm of inverse error propagation and the stochastic method of particle swarm for training neural networks of direct propagation, their advantages and disadvantages are indicated. It is proposed to combine the advantages of both methods in a combined algorithm.The learning process using a combined algorithm is carried out in two stages. At the first stage, the stochastic method of particle swarm conducts a given number of learning epochs and determines the set of points in the vicinity of which there may be points of local minimum. In the second stage, the gradient backpropagation algorithm finds the local minimum for each point and selects the optimal one. If the set value of the standard error of the output is not reached, the learning steps are repeated.To evaluate the effectiveness of the proposed approach to the training of neural networks, a series of comparative experiments using the well-known database of computer attack recognition KDD Cup 1999 Data. The experiments compared the results of training the direct propagation neural network for the particle swarm method, the inverse error propagation algorithm, and the combined algorithm. The experimental results proved the superiority of the combined algorithm.