Words matter: using natural language processing to predict neurosurgical residency match outcomes

Abstract

OBJECTIVE Narrative letters of recommendation (NLORs) are considered by neurosurgical program directors to be among the most important parts of the residency application. However, the utility of these NLORs in predicting match outcomes compared to objective measures has not been determined. In this study, the authors compare the performance of machine learning models trained on applicant NLORs and demographic data to predict match outcomes and investigate whether narrative language is predictive of standardized letter of recommendation (SLOR) rankings. METHODS This study analyzed 1498 NLORs from 391 applications submitted to a single neurosurgery residency program over the 2020–2021 cycle. Applicant demographics and match outcomes were extracted from Electronic Residency Application Service applications and training program websites. Logistic regression models using least absolute shrinkage and selection operator were trained to predict match outcomes using applicant NLOR text and demographics. Another model was trained on NLOR text to predict SLOR rankings. Model performance was estimated using area under the curve (AUC). RESULTS Both the NLOR and demographics models were able to discriminate similarly between match outcomes (AUCs 0.75 and 0.80; p = 0.13). Words including “outstanding,” “seamlessly,” and “AOA” (Alpha Omega Alpha) were predictive of match success. This model was able to predict SLORs ranked in the top 5%. Words including “highest,” “outstanding,” and “best” were predictive of the top 5% SLORs. CONCLUSIONS NLORs and demographic data similarly discriminate whether applicants will or will not match into a neurosurgical residency program. However, NLORs potentially provide further insight regarding applicant fit. Because words used in NLORs are predictive of both match outcomes and SLOR rankings, continuing to include narrative evaluations may be invaluable to the match process.