Improving Case Duration Accuracy of Orthopedic Surgery Using Bidirectional Encoder Representations from Transformers (BERT) on Radiology Reports

Abstract

Abstract Purpose: A major source of inefficiency in the operating room is the mismatch between scheduled versus actual surgical time. The purpose of this study was to demonstrate a proof-of-concept study for predicting case duration by applying natural language processing (NLP) and machine learning that interpret radiology reports for patients undergoing radius fracture repair. Methods: Logistic regression, random forest, and artificial neural networks (ANN) were tested without NLP and with bag-of-words. Another NLP method tested used ANN and Bidirectional Encoder Representations from Transformers specifically pre-trained on clinical notes (ClinicalBERT). A total of 201 cases were included. The data were split into 70% training and 30% test sets. The average root mean squared error (RMSE) (and 95% confidence interval [CI]) from 10-fold cross-validation on the training set were used to develop each model. Models were then compared to a baseline model, which used historic averages to predict surgical time. Results: The average RMSE was lowest using ANN with ClinicalBERT (25.6 minutes, 95% CI: 21.5 - 29.7), which was significantly (P<0.001) lower than the baseline model (39.3 minutes, 95% CI: 30.9 - 47.7). Using the ANN and ClinicalBERT on the test set, the percentage of accurately predicted cases, which was defined by the actual surgical duration within 15% of the predicted surgical duration, increased from 26.8% to 58.9% (P<0.001). Conclusion: This proof-of-concept study demonstrated the successful application of NLP and machine leaning to extract features from unstructured clinical data resulting in improved prediction accuracy for surgical case duration.