Abstract
ABSTRACTIn the context of Artificial Intelligence (AI)-driven decision support systems for high-stakes environments, particularly in healthcare, ensuring the safety of human-AI interactions is paramount, given the potential risks associated with erroneous AI outputs. To address this, we conducted a prospective observational study involving 38 intensivists in a simulated medical setting.Physicians wore eye-tracking glasses and received AI-generated treatment recommendations, including unsafe ones. Most clinicians promptly rejected unsafe AI recommendations, with many seeking senior assistance. Intriguingly, physicians paid increased attention to unsafe AI recommendations, as indicated by eye-tracking data. However, they did not rely on traditional clinical sources for validation post-AI interaction, suggesting limited “debugging.”Our study emphasises the importance of human oversight in critical domains and highlights the value of eye-tracking in evaluating human-AI dynamics. Additionally, we observed human-human interactions, where an experimenter played the role of a bedside nurse, influencing a few physicians to accept unsafe AI recommendations. This underscores the complexity of trying to predict behavioural dynamics between humans and AI in high-stakes settings.
Publisher
Cold Spring Harbor Laboratory
Reference38 articles.
1. Wang G , Liu X , Ying Z , Yang G , Chen Z , Liu Z , et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial. Nat Med. 2023 Sep 14;1–10.
2. DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning;Proc AAAI Conf Artif Intell,2022
3. Nagendran M , Festor P , Komorowski M , Gordon A , Faisal AA . Quantifying the impact of AI recommendations with explanations on prescription decision making: an interactive vignette study [Internet]. 2023 [cited 2023 Jun 19]. Available from: https://www.researchsquare.com
4. Bad machines corrupt good morals;Nat Hum Behav,2021
5. Festor P , Habli I , Jia Y , Gordon A , Faisal AA , Komorowski M . Levels of Autonomy and Safety Assurance for AI-Based Clinical Decision Systems. In Springer; 2021. p. 291–6.