Visual Emotion Analysis via Affective Semantic Concept Discovery

Abstract

With the development of social media, people prefer to express views and share daily life online via visual content, which has led to widespread attention in automatic emotion analysis from images. Capturing the emotions embedded in these social images has always been important yet challenging. In this paper, we propose a visual emotion prediction method that utilizes the affective semantic concepts of an image to predict its emotion. To solve the problems of narrow semantic coverage and low discriminative power of emotions in current semantic concept sets used for visual emotion analysis, we develop a concept selection model to mine emotion-related concepts from social media. Specifically, we propose several selection strategies to build an affective semantic concept set that contains various visual concepts related to emotion conveyance. And they are discovered from affective image datasets and associated tags crawled from websites. To further leverage the discovered affective semantic concepts, we train concept classifiers to predict the concept score of each concept, which are used as the intermediate features to tackle the semantic gap problem for image emotion recognition. Extensive experimental results confirm the validity of the affective semantic concepts and show the improved performance of our method.