User Experience Quantification Model from Online User Reviews

Abstract

Due to the advancement in information technology and the boom of micro-blogging platforms, a growing number of online reviews are posted daily on product distributed platforms in the form of spontaneous and insightful user feedback, and these can be used as a significant data source to understand user experience (UX) and satisfaction. However, despite the vast amount of online reviews, the existing literature focuses on online ratings and ignores the real textual context in reviews. We proposed a three-step UX quantification model from online reviews to understand customer satisfaction using the effect-based Kano model. First, the relevant online reviews are selected using various filter mechanisms. Second, UX dimensions (UXDs) are extracted using a proposed method called UX word embedding Latent Dirichlet allocation (UXWE-LDA) and sentiment orientation using a transformer-based pipeline. Then, the casual relationships are identified for the extracted UXDs. Third, the UXDs are mapped on the customer satisfaction model (effect-based Kano) to understand the user perspective about the system, product, or services. Finally, the different parts of the proposed quantification model are evaluated to examine the performance of this method. We present different results of the proposed method in terms of accuracy, topic coherence (TC), Topic-wise performance, and expert-based evaluation for the proposed framework validation. For review quality filters, we achieved 98.49% accuracy for the spam detection classifier and 95% accuracy for the relatedness detection classifier. The results show that the proposed method for the topic extractor module always gives a higher TC value than other models such as WE-LDA and LDA. Regarding topic-wise performance measures, UXWE-LDA achieves a 3% improvement on average compared to LDA due to the incorporation of semantic domain knowledge. We also compute the Jaccard coefficient similarity between the extracted dimensions using UXWE-LDA and UX experts-based analysis for checking the mutual agreement, which is 0.3, 0.5, and 0.4, respectively. Based on the Kano model, the presented study has potential implications concerning issues and knowing the product’s strengths and weaknesses in product design.