Controlling Safety of Artificial Intelligence-Based Systems in Healthcare

Abstract

Artificial intelligence (AI)-based systems have achieved significant success in healthcare since 2016, and AI models have accomplished medical tasks, at or above the performance levels of humans. Despite these achievements, various challenges exist in the application of AI in healthcare. One of the main challenges is safety, which is related to unsafe and incorrect actions and recommendations by AI algorithms. In response to the need to address the safety challenges, this research aimed to develop a safety controlling system (SCS) framework to reduce the risk of potential healthcare-related incidents. The framework was developed by adopting the multi-attribute value model approach (MAVT), which comprises four symmetrical parts: extracting attributes, generating weights for the attributes, developing a rating scale, and finalizing the system. The framework represents a set of attributes in different layers and can be used as a checklist in healthcare institutions with implemented AI models. Having these attributes in healthcare systems will lead to high scores in the SCS, which indicates safe application of AI models. The proposed framework provides a basis for implementing and monitoring safety legislation, identifying the risks in AI models’ activities, improving human-AI interactions, preventing incidents from occurring, and having an emergency plan for remaining risks.