OR Vision: Objective, explainable assessment of surgical skill with deep learning

Abstract

Abstract Background Currently, evaluating surgical technical performance is inefficient and subjective [1,2,3,4] and the established rubrics for assessing surgical ability are open to interpretation. To power programs for surgical training and Maintenance of Certification (MOC), a reliable and validated solution is required. To this end, we draw upon recent advances in machine learning and propose a framework for objective and scalable assessment of technical proficiency.Methods Different machine learning models were trained to predict surgical performance on the public EndoVis19 and JIGSAWS datasets. The most important features were extracted by probing each machine learning model, and these features form the basis of the proposed algorithm. We internally tested the performance of this model on proprietary datasets from Surgical Safety Technologies (SST) and the University of Texas Southwestern (UTSW). The performance of these models was assessed according to various statistical techniques such as precision, recall, f1-scores and the area under the receiver operating characteristic curve (AUC). Results OR Vision is a statistically-driven multi-stage machine learning tool that quantifies surgical skill objectively and explainably. Instrument motion, control, and coordination are quantified in terms of 150 objective metrics, extracted from tool motion tracked by the deep learning model. The N most highly correlated of these metrics (p<0.05) model surgical performance with quantifiable objective metrics (fine-motor precision, fluidity, tremor, disorder, etc.). These metrics are combined into clinically-weighted composite scores that represent the category-wise technical performance of surgeons. The OR Vision score discriminates between expert and novice surgeons with high precision (0.82-0.84) and provides constructive feedback in the form of a concise report for every participating member of the cohort. Each report provides a breakdown of user performance on statistically relevant categories.ConclusionA machine learning-based approach for identifying surgical skill is effective and meaningful and provides the groundwork for objective, precise, repeatable, cost-effective, clinically-meaningful assessments.