A THEORETICAL INVESTIGATION ON TRAINING OF PIPE-LIKE NEURAL NETWORK BENCHMARK ARCHITECTURES AND PERFORMANCE COMPARISONS OF POPULAR TRAINING ALGORITHMS

Abstract

Architectures of neural networks affect the training performance of artificial neural networks. For more consistent performance evaluation of training algorithms, hard-to-train benchmarking architectures should be used. This study introduces a benchmark neural network architecture, which is called pipe-like architecture, and presents training performance analyses for popular Neural Network Backpropagation Algorithms (NNBA) and well-known Metaheuristic Search Algorithms (MSA). The pipe-like neural architectures essentially resemble an elongated fraction of a deep neural network and form a narrowed long bottleneck for the learning process. Therefore, they can significantly complicate the training process by causing the gradient vanishing problems and large training delays in backward propagation of parameter updates throughout the elongated pipe-like network. The training difficulties of pipe-like architectures are theoretically demonstrated in this study by considering the upper bound of weight updates according to an aggregated one-neuron learning channels conjecture. These analyses also contribute to Baldi et al.'s learning channel theorem of neural networks in a practical aspect. The training experiments for popular NNBA and MSA algorithms were conducted on the pipe-like benchmark architecture by using a biological dataset. Moreover, a Normalized Overall Performance Scoring (NOPS) was performed for the criterion-based assessment of overall performance of training algorithms.