Abstract
The activation function plays an important role in training and improving performance in deep neural networks (dnn). The rectified linear unit (relu) function provides the necessary non-linear properties in the deep neural network (dnn). However, few papers sort out and compare various relu activation functions. Most of the paper focuses on the efficiency and accuracy of certain activation functions used by the model, but does not pay attention to the nature and differences of these activation functions. Therefore, this paper attempts to organize the RELU-function and derived function in this paper. And compared the accuracy of different relu functions (and its derivative functions) under the Mnist data set. From the experimental point of view, the relu function performs the best, and the selu and elu functions perform poorly.
Reference29 articles.
1. Nair V. & Hinton G. E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. In Fürnkranz J. & Joachims T. (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10) (p./pp. 807-814).
2. Weight Initialization Possibilities for Feedforward Neural Network with Linear Saturated Activation Functions
3. Clevert D. A., Unterthiner T., & Hochreiter S. (2015). Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289.
Cited by
27 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献