Determining the best feature combination through text and probabilistic feature analysis for GPT-2-based mobile app review detection

Abstract

AbstractMobile apps, used by many people worldwide, have become an essential part of life. Before using a mobile app, users judge the reliability of apps according to their reviews. Therefore, app reviews are essential components of management for companies. Unfortunately, some fake reviewers write negative reviews for competing apps. Moreover, artificial intelligence (AI)-based macro bot programs that generate app reviews have emerged and can create large numbers of reviews with malicious purposes in a short time. One notable AI technology that can generate such reviews is Generative Pre-trained Transformer-2 (GPT-2). The reviews generated by GPT-2 use human-like grammar; therefore, it is difficult to detect them with only text mining techniques, which use tools like part-of-speech (POS) tagging and sentiment scores. Thus, probability-based sampling techniques in GPT-2 must be used. In this study, we identified features to detect reviews generated by GPT-2 and determined the optimal feature combination for improving detection performance. To achieve this, based on the analysis results, we built a training dataset to find the best feature combination for detecting the generated reviews. Various machine learning models were then trained and evaluated using this dataset. As a result, the model that used both text mining and probability-based sampling techniques detected generated reviews more effectively than the model that used only text mining techniques. This model achieved a top classification accuracy of 90% and a macro F1 of 0.90. We expect the results of this study to help app developers maintain a more stable mobile app ecosystem. Graphical abstract