Commercial chatbot monitoring: Approaches focused on automated conversation analysis

Abstract

Purpose of the study: The purpose of this study is to review and analyze current automated techniques for monitoring chatbot conversations in the field of Conversational Artificial Intelligence. It aims to highlight the challenges and limitations of these techniques and provide insights into various metrics used to measure chatbot performance, with the goal of enhancing it. Methodology: The study employs a comprehensive literature review of existing automated techniques for monitoring chatbot conversations. Then, focusing on state-of-the-art approaches, the study introduces a division into numerical metrics (performance statistics and user engagement) and linguistic metrics (conversation analysis). Within conversation analysis, which is crucial for improving chatbot responses and accurately recognizing user intentions, the study identifies and presents three leading methods. Main Findings: The paper highlights that, while current chatbot numerical conversation metrics allow for continuous monitoring and enhancement of chatbot performance, there is still room for improvement in the automated linguistic analysis of chatbot conversations. Furthermore, monitoring chatbot conversations in an automatic way in order to implement adequate corrective actions, is an essential task for refining chatbot efficiency through continuous learning and adaptation. Applications of the study: The findings of this study have practical applications for businesses employing chatbots. By understanding the potential of current automated monitoring techniques and addressing their limitations, commercial chatbot systems can be improved for the benefit of customer satisfaction. Novelty/Originality of the study: The paper provides readers with the novel knowledge necessary to understand key metrics used to measure chatbot conversations from both numerical and linguistic perspectives. It adds value by guiding readers on how monitoring numerical metrics helps analyze chatbot interactions and explains how the automated linguistic analysis of chatbot conversation content is utilized in leading approaches.