Clustering Methods and Tools to Handle High-Dimensional Social Media Text Data

Abstract

Social media data has changed the way big data is used. The amount of data available offers more natural insights that make it possible to find relations and social interactions. Natural language processing (NLP) is an essential tool for such a task. NLP promises to scale traditional methods that allow the automation of tasks for social media datasets. A social media text dataset with a large number of attributes is referred to as a high-dimensional text dataset. One of the challenges of high-dimensional text datasets for NLP text clustering is that not all the measured variables are important for understanding the underlying phenomena of interest, and dimension reduction needs to be performed. Nonetheless, for text clustering, the existing literature is remarkably segmented, and the well-known methods do not address the problems of the high dimensionality of text data. Thus, different methods were found and classified in four areas. Also, it described metrics and technical tools as well as future directions.