Evaluation Techniques for Chatbot Usability: A Systematic Mapping Study

Abstract

Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state-of-the-art in chatbot usability and applied human–computer interaction techniques, to analyze how to evaluate chatbot usability. Method: We have conducted a systematic mapping study, by searching the main scientific databases. The search retrieved 170 references and 21 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is still a very incipient field of research where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.