Real-time Emotion Detection by Quantitative Facial Motion Analysis

Abstract

AbstractBackgroundResearch into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools.MethodsTo address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis.ResultsWe identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias.LimitationsOur sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals.ConclusionsWe demonstrate that DISC-based facial analysis can be used to reliably identify an individual’s emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.HighlightsWe detected the emotional state of participants in the absence of visible changes in facial expressionsWe identified distinct facial movement maps for happy and sad emotionsWe developed a highly sensitive, noninvasive, noncontact emotional state assessment tool