Redis-Based Messaging Queue and Cache-Enabled Parallel Processing Social Media Analytics Framework

Abstract

Abstract The extensive usage of social media polarity analysis claims the need for real-time analytics and runtime outcomes on dashboards. In data analytics, only 30% of the time is consumed in modeling and evaluation stages and 70% is consumed in data engineering tasks. There are lots of machine learning algorithms to achieve a desirable outcome in prediction points of view, but they lack in handling data and their transformation so-called data engineering tasks, and reducing its time remained still challenging. The contribution of this research paper is to encounter the mentioned challenges by presenting a parallelly, scalable, effective, responsive and fault-tolerant framework to perform end-to-end data analytics tasks in real-time and batch-processing manner. An experimental analysis on Twitter posts supported the claims and signifies the benefits of parallelism of data processing units. This research has highlighted the importance of processing mentioned URLs and embedded images along with post content to boost the prediction efficiency. Furthermore, this research additionally provided a comparison of naive Bayes, support vector machines, extreme gradient boosting and long short-term memory (LSTM) machine learning techniques for sentiment analysis on Twitter posts and concluded LSTM as the most effective technique in this regard.