Investigating Fairness in Machine Learning-based Audio Sentiment Analysis using Spectrograms and Bag-of-visual-words

Abstract

Abstract Audio sentiment analysis is a growing area of research, however fairness in audio sentiment analysis is hardly investigated. We found research on machine learning tools’ reliability and fairness in various demographic groups. However, fairness in audio sentiment analysis regarding gender is still an uninvestigated field. In this research, we used 442 audio files of happiness and sadness -- representing equal samples of male and female subjects -- and generated spectrograms for each file. Then we used bag-of-visual-words method for feature extraction and Random Forest, Support Vector Machines and K-nearest Neighbors classifiers to investigate whether the machine learning models for audio sentiment analysis are fair among the two genders. We found the need for gender-specific models for audio sentiment analysis instead of a gender-agnostic general-model. Our results provided three pieces of evidence to back up our claim that the gender-agnostic model is bias in terms of accuracy of the audio sentiment analysis task. Furthermore, we discovered that a gender-specific model trained with female audio samples does not perform well against male audio files and vice versa. The best accuracy for female-model is 76% and male-model is 74%, which is significantly better than the gender-agnostic model’s accuracy of 66%.