Managing risk and resilience in autonomous and intelligent systems: Exploring safety in the development, deployment, and use of artificial intelligence in healthcare

Abstract

AbstractAutonomous and intelligent systems (AIS) are being developed and deployed across a wide range of sectors and encompass a variety of technologies designed to engage in different forms of independent reasoning and self‐directed behavior. These technologies may bring considerable benefits to society but also pose a range of risk management challenges, particularly when deployed in safety‐critical sectors where complex interactions between human, social, and technical processes underpin safety and resilience. Healthcare is one safety‐critical sector at the forefront of efforts to develop and deploy intelligent technologies, such as through artificial intelligence (AI) systems intended to automate key aspects of healthcare tasks such as reading medical images to identify signs of pathology. This article develops a qualitative analysis of the sociotechnical sources of risk and resilience associated with the development, deployment, and use of AI in healthcare, drawing on 40 in‐depth interviews with participants involved in the development, management, and regulation of AI. Qualitative template analysis is used to examine sociotechnical sources of risk and resilience, drawing on and elaborating Macrae's (2022, Risk Analysis, 42(9), 1999–2025) SOTEC framework that integrates structural, organizational, technological, epistemic, and cultural sources of risk in AIS. This analysis explores an array of sociotechnical sources of risk associated with the development, deployment, and use of AI in healthcare and identifies an array of sociotechnical patterns of resilience that may counter those risks. In doing so, the SOTEC framework is elaborated and translated to define key sources of both risk and resilience in AIS.