An evaluation of ChatGPT compared with dermatological surgeons’ choices of reconstruction for surgical defects after Mohs surgery

Abstract

Abstract Background ChatGPT is an open-access chatbot developed using artificial intelligence (AI) that generates human-like responses. Objective To evaluate the ChatGPT-4’s concordance with three dermatological surgeons on reconstructions for dermatological surgical defects. Methods The cases of 70 patients with nonmelanoma skin cancer treated with surgery were obtained from clinical records for analysis. A list of 30 reconstruction options was designed by the main authors that included primary closure, secondary skin closure, skin flaps and skin grafts. Three dermatological surgeons who were blinded to the real reconstruction, along with ChatGPT-4, were asked to select two reconstruction options from the list. Results Seventy responses were analysed using Cohen’s kappa, looking for concordance between each dermatologist and ChatGPT. The level of agreement among dermatological surgeons was higher compared with that between dermatological surgeons and ChatGPT, highlighting differences in decision making. In the selection of the best reconstruction technique, the results indicated a fair level of agreement among the dermatologists, ranging between κ 0.268 and 0.331. However, the concordance between ChatGPT-4 and the dermatologists was slight, with κ values ranging from 0.107 to 0.121. In the analysis of the second-choice options, the dermatologists showed only slight agreement. In contrast, the level of concordance between ChatGPT-4 and the dermatologists was below chance. Conclusions As anticipated, this study reveals variability in medical decisions between dermatological surgeons and ChatGPT. Although these tools offer exciting possibilities for the future, it is vital to acknowledge the risk of inadvertently relying on noncertified AI for medical advice.