SentiNet: A Nonverbal Facial Sentiment Analysis Using Convolutional Neural Network

Abstract

Human facial expressions are an essential and fundamental component for expressing the state of the human mind. The automatic analysis of these nonverbal facial expressions has become a fascinating and quite challenging problem in computer vision, with its application in different areas, such as psychology, human–machine interaction, health, and augmented reality. Recently, deep learning (DL) has become a widespread technique for studying human nonverbal facial sentiment expressions, and some research attempts have been made to propose a certain model on this topic. The purpose of this paper is to apply the appropriate convolutional neural network (CNN) approach by adding several layers of different dimensions, which allows the CNN approach to efficiently classify human facial sentiment expressions with data augmentation capable of recognizing seven basic human facial expressions: anger, sadness, fear, disgust, happiness, surprise, and neutral. In particular, this study mainly proposes a convolution neural network architecture, as well as learning factors that minimize the memory space and total training time of the proposed network due to the shallow architecture of the model. Following that, we demonstrated our proposed model’s network complexity, computational cost, and classification accuracy on the three benchmark datasets: FER2013, KDEF, and JAFFE. As a result, our proposed approach achieves accuracy of [Formula: see text], [Formula: see text], [Formula: see text] in the FER2013, KDEF, and JAFFE, respectively, which is better compared to other state-of-the-art approaches.