How Does Variation in AI Performance Affect Trust in AI-infused Systems: A Case Study With In-Vehicle Voice Control Systems

Abstract

This study investigates how performance variation in AI-infused systems, specifically in-vehicle voice control systems, affects user trust. The frequent updates of AI-infused systems over the air raise concerns about the unstable performance of AI-infused systems, due to uncertainty and inexplicability of AIs. Employing a Wizard of Oz simulation with 27 participants, the study manipulated the accurate correct rates of voice command responses to represent system updates, measuring the perceived correct rate and trust after each update. Results suggest that the way the system evolves can impact users’ trust in the current version of the system, but this effect fades out with time. Moreover, users’ perception of performance, rather than the actual performance, has an impact on users’ trust. The research underscores the importance of managing users’ perception of system performance to maintain user trust and contributes to the knowledge of the relationship between AI evolution and user trust.