Multimodal Depression Detection: Using Fusion Strategies with Smart Phone Usage and Audio-visual Behavior

Abstract

The problem of detecting depression is multi-faceted because of variability in depressive symptoms caused by individual differences. The variations can be seen in historical information (like decreased physical activity etc.) and also in verbal/non-verbal behaviors (like lower pitch, downward eye gaze etc.). The primary goal of this research is to develop a novel classification system for diagnosing depression that considers both historical information and also verbal/non-verbal behaviors. For this purpose, we created a realworld multimodal dataset of depressed and non-depressed subjects with fourteen-day real-time smartphone usage records and audio-visual recordings. We extracted numerous features related to physiological/physical activity from smartphone usage records to capture historical information and features like pitch and eye gaze (verbal and non-verbal manifestations) from audio-visual clues. We experimented with early fusion using Decision trees classifier (along with several feature selection strategies) and Support Vector Machine (SVM) classifier with several late fusion methods. Then, we conducted a comparative study among both fusion strategies. Our findings showed that SVM classifier using late fusion strategy achieves best accuracy of 89%. In addition, a popular benchmarking multimodal dataset (DAIC-WOZ database) is used to further validate the effectiveness of our approach by fusing multi-faceted feature vectors for depression detection.