Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment: Analyzing the Sentiment Bias of Four Major Tools-Reference-Cited by-同舟云学术

Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment: Analyzing the Sentiment Bias of Four Major Tools

Published:2022-06-14 Issue:EICS Volume:6 Page:1-20
ISSN:2573-0142
Container-title:Proceedings of the ACM on Human-Computer Interaction
language:en
Short-container-title:Proc. ACM Hum.-Comput. Interact.

Author:

Jung Soon-Gyo¹,Salminen Joni²,Jansen Bernard J.¹

Affiliation:

1. Hamad bin Khalifa University, Doha, Qatar

2. University of Vaasa, Vaasa, Finland

Abstract

Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these tools are, and whether the sentiment ratings given by one tool agree with those given by another tool. We use two datasets - (1) NEWS consisting of 5,880 news stories and 60K comments from four social media platforms: Twitter, Instagram, YouTube, and Facebook; and (2) IMDB consisting of 7,500 positive and 7,500 negative movie reviews - to investigate the agreement and bias of four widely used sentiment analysis (SA) tools: Microsoft Azure (MS), IBM Watson, Google Cloud, and Amazon Web Services (AWS). We find that the four tools assign the same sentiment on less than half (48.1%) of the analyzed content. We also find that AWS exhibits neutrality bias in both datasets, Google exhibits bi-polarity bias in the NEWS dataset but neutrality bias in the IMDB dataset, and IBM and MS exhibit no clear bias in the NEWS dataset but have bi-polarity bias in the IMDB dataset. Overall, IBM has the highest accuracy relative to the known ground truth in the IMDB dataset. Findings indicate that psycholinguistic features - especially affect, tone, and use of adjectives - explain why the tools disagree. Engineers are urged caution when implementing SA tools for applications, as the tool selection affects the obtained sentiment labels.

Publisher

Association for Computing Machinery (ACM)

Subject

Computer Networks and Communications,Human-Computer Interaction,Social Sciences (miscellaneous)

Link

https://dl.acm.org/doi/pdf/10.1145/3532203

Reference57 articles.

1. Moloud Abdar , Farhad Pourpanah , Sadiq Hussain , Dana Rezazadegan , Li Liu , Mohammad Ghavamzadeh , Paul Fieguth , Xiaochun Cao , Abbas Khosravi , and U. Rajendra Acharya . 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. (2021) . Publisher : Elsevier . Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, and U. Rajendra Acharya. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. (2021). Publisher: Elsevier.

2. AWATIF: A Multi-Genre Corpus for Modern Standard Arabic Subjectivity and Sentiment Analysis;Abdul-Mageed Muhammad;LREC,2012

3. Kholoud Khalil Aldous , Jisun An , and Bernard J Jansen . 2019 . View, Like, Comment, Post: Analyzing User Engagement by Topic at 4 Levels Across 5 Social Media Platforms for 53 News Organizations . In Proceedings of the International AAAI Conference on Web and Social Media. Kholoud Khalil Aldous, Jisun An, and Bernard J Jansen. 2019. View, Like, Comment, Post: Analyzing User Engagement by Topic at 4 Levels Across 5 Social Media Platforms for 53 News Organizations. In Proceedings of the International AAAI Conference on Web and Social Media.

4. Approaches, tools and applications for sentiment analysis implementation;Alessia D.;International Journal of Computer Applications,2015

5. Debugging a Crowdsourced Task with Low Inter-Rater Agreement

Cited by 6 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis;Information Processing & Management;2023-11

2. Handwriting Analysis on the Diaries of Rosamond Jacob;20th International Conference on Content-based Multimedia Indexing;2023-09-20

3. Social Media Analytics;Understanding Audiences, Customers, and Users via Analytics;2023-09-06

4. The Foundations of Social Media Analytics;Understanding Audiences, Customers, and Users via Analytics;2023-09-06

5. A Discussion of the Validity of Data Analytics;Understanding Audiences, Customers, and Users via Analytics;2023-09-06