A Systematic Mapping Study about Technologies for Hedonic Aspects Evaluation in Text-based Chatbots

Abstract

Many studies present and evaluate daily-use systems ranging from information to conversational systems. Chatbots, either text-based or voice-based, have attracted the attention of researchers. In particular, User eXperience (UX) has been pointed out as one of the chatbot's leading aspects of evaluation involving pragmatic and hedonic aspects. Pragmatic aspects deal with the usability and efficiency of the system, while hedonic aspects consider aspects related to the originality, innovation, beauty of the system, and the user's psychological well-being. Even with existing research on usability evaluation and human-computer interaction within conversational systems, there is a clear shortfall in studies specifically addressing the hedonic aspects of user experience in chatbots. Therefore, this paper presents a Systematic Mapping Study that investigates various UX evaluation technologies (questionnaires, methods, techniques, and models, among others), focusing on the hedonic aspect of chatbots. We focused on studies with chatbots that are activated by text, although they may be able to display click interactions, videos, and images in addition to the text modality. We discovered 69 technologies to evaluate hedonic aspects of UX in chatbots, and the most frequent aspect found is the General UX . Our study provides relevant data on the research topic, addressing the specific characteristics of human-chatbot interaction, such as identity and social interaction. Moreover, we highlight gaps in the hedonic aspect evaluation in chatbots, such as a few works investigating the assessment of user emotional state.