Emotion identification in human faces through ensemble of deep learning models

Abstract

Investigation of human face images forms an important facet in affective analysis. The work, a DL-based ensemble is proposed for this purpose. Seven pre-trained models namely Facenet, Facenet2018, VGG16, Resnet-50, Senet-50, Arcface and Openface that have been developed for face verification have been exploited and customized for emotion identification. To each of these models, each all over interaction with softmax method to classification groups are augmented and entire network is then trained completely for emotion recognition. After training all the models individually, the probabilities for each of the class by each of the model are summed to derive at the final value. The class that holds the highest of this value is finalized as the predicted emotion. Thus, the proposed methodology involves image collection, image pre-processing comprising of contrast enhancement, face detection and extraction, face alignment, image augmentation facilitating rotation, shifting, flipping and zooming transformations and appropriate resizing and rescaling, feature extraction and classification through ensemble of customized afore-mentioned pre-trained convolutional neural networks, evaluation and evolving of best weights for emotion recognition from face images with enhanced accuracy. The proposed methodology is evaluated on the well-established FER-2013 dataset. The methodology achieves a validation accuracy of 74.67% and test accuracy of 76.23%. Further, similar images of another dataset (Face Expression Recogniton dataset) are included for training the models and the impact of extra training is assessed to see if there is improvement in performance. The experiments reveal marked improvement in face emotion identification performance reaching values of 94.98% for both validation and test set of FER-2013 dataset and 94.99% on validation set of Face Expression Recognition dataset.