The performance of ChatGPT in day surgery and pre-anesthesia risk assessment: a case-control study across on 150 simulated patient presentations

Abstract

Background Day surgery has developed rapidly in China in recent years, although it still faces the shortage of anesthesiologists to handle pre-anesthesia routine for day surgery. We hypothesized that Chat Generative Pre-trained Transformer (ChatGPT) may assist anesthesiologists in preoperative assessment and answer questions on the patients' concerns. The aim of this study was to examine the ability of ChatGPT in pre-operative risk assessment and its accuracy to answer questions regarding knowledge and management for day surgery anesthesia. Methods 150 patient profiles were generated to simulate day surgery patient presentations, with complications of varying acuity and severity. Both ChatGPT group and expert group has to evaluate 150 patients for their ASA classification and whether day surgery was recommended. Then ChatGPT was asked to answer 145 questions about day surgery anesthesia that represented the most common issues encountered in clinical practice. The performance of ChatGPT was assessed and graded independently by two experienced anesthesiologists. Results A total of 150 patients for assessment were included in the study (74 males [49.5%] and 76 females [50.7%]). There was no difference between ChatGPT group and the expert group for ASA classification and assessment of anesthesia risk in the patients (P > 0.05). When comes to day surgery recommendation, for patients with certain comorbidities (ASA ≥ II), the expert group was inclined to need further examination or treatment, the proportion of conclusions made by ChatGPT was smaller (ChatGPT n (%) vs. expert n (%): day surgery can be performed, 67 (47.9) vs. 31(25.4); need further treatment and evaluation, 56 (37.3) vs. 66 (44.0); day surgery is not recommended, 18 (12.9) vs. 29 (9.3), P < 0.05). We showed that ChatGPT had extensive knowledge related to day surgery anesthesia (94.0% correct), with most of the points (70%) considered comprehensive. The performance was better in the domains of peri-anesthesia concerns, lifestyle, and emotional support. Conclusions ChatGPT has certain reference value for ASA classification and assessment of anesthesia risk in patients. ChatGPT can also be trusted to answer questions and concerns related to pre-anesthesia and therefore has the potential to provide important assistance in clinical work.