Abstract
Machine learning tools are increasingly used to improve the quality of care and the soundness of a treatment plan. Explainable AI (XAI) helps users in understanding the inner mechanisms of opaque machine learning models and is a driver of trust and adoption. Explanation methods for black-box models exist, but there is a lack of user studies on the interpretability of the provided explanations. We used a Think Aloud Protocol (TAP) to explore oncologists’ assessment of a lung cancer relapse prediction system with the aim of refining the purpose-built explanation model for better credibility and utility. Novel to this context, TAP is used as a neutral methodology to elicit experts’ thought processes and judgements of the AI system, without explicit prompts. TAP aims to elicit the factors which influenced clinicians’ perception of credibility and usefulness of the system. Ten oncologists took part in the study. We conducted a thematic analysis of their verbalized responses, generating five themes that help us to understand the context within which oncologists’ may (or may not) integrate an explainable AI system into their working day.
Publisher
Public Library of Science (PLoS)
Reference43 articles.
1. Brilliant AI Doctor” in Rural Clinics: Challenges in AI-Powered Clinical Decision Support System Deployment;D. Wang;in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems,2021
2. The stakes of uncertainty: developing and integrating machine learning in clinical care.;M.C. Elish;in Ethnographic Praxis in Industry Conference Proceedings.,2018
3. Machine learning models for early sepsis recognition in the neonatal intensive care unit using readily available electronic health record data;A.J. Masino;PLoS One,2019
4. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death;G.B. Smith;Resuscitation,2013
5. Think-aloud technique and protocol analysis in clinical decision-making research.;H. Lundgrén-Laine;Qualitative health research,,2010