Monitoring performance of clinical artificial intelligence: a scoping review protocol

Abstract

Objective: The objective of this scoping review is to elucidate the scope and nature of research on the monitoring of clinical artificial intelligence (AI) systems. The review will identify the various methodologies used to monitor clinical AI, while also mapping the reasons that influence the selection of monitoring approaches. Introduction: AI is being used in clinical decision-making at an increasing rate. While much attention has been directed toward the development and validation of AI for clinical applications, the practical implementation aspects, notably the establishment of rational monitoring/quality assurance systems, has received comparatively limited scientific interest. Given the scarcity of evidence and the heterogeneity of methodologies used in this domain, there is a compelling rationale for conducting a scoping review on this subject. Inclusion criteria: This scoping review will include any publications that describe systematic, continuous, or repeated initiatives that evaluate or predict clinical performance of AI models with direct implications for the management of patients in any segment of the health care system. Methods: Publications will be identified through searches of the MEDLINE (Ovid), Embase (Ovid), and Scopus databases. Additionally, backward and forward citation searches as well as a thorough investigation of gray literature will be conducted. Title and abstract screening, full-text evaluation, and data extraction will be performed by 2 or more independent reviewers. Data will be extracted using a tool developed by the authors. The results will be presented graphically and narratively. Review registration: Open Science Framework https://osf.io/afkrn