Quality over quantity? The role of data quality and uncertainty for AI in surgery

Abstract

AbstractSurgical Data Science is an emerging scientific discipline that applies advances in data science, machine learning and AI to harness the increasingly large amounts of surgical data to enable AI in surgery [1–4]. Data collection for AI solutions involves both ingestion of contingent data (in case of surgery—medical records, case data, instrument data, medical images, data from OR sensors and surgical video), as well as intentionally collected annotations and expert opinion describing the data. This organized knowledge is then used to train AI models that ultimately generate predictions based on the available training data. Historically, the data science workflow starts with organizing a clean and consistent dataset, with the mantra GIGO—garbage in, garbage out—emphasizing that the quality of the model output is directly related to the quality of data. In surgery, as in healthcare in general, this is not an easy goal to achieve due to the complex logistics of data collection, missing and incomplete data, human error, the lack of measurement standards, as well as subjective differences in data interpretation. In this article, we look at surgical AI from this particular perspective of data quality and uncertainty. We highlight a few topics of which hospitals, surgeons and research teams need to be aware when collecting data for AI that will provide actionable outputs in clinical and educational settings.