Does the Medical Student Performance Evaluation Change the Decision to Invite Residency Applicants?

Abstract

Introduction: Although emergency medicine (EM) residency program directors (PD) have multiple sources to evaluate each applicant, some programs await the release of the medical student performance evaluation (MSPE) to extend interview offers. While prior studies have demonstrated that MSPE content is variable and selectively positive, no prior work has evaluated the impact of the MSPE on the likelihood to invite (LTI) applicants for a residency interview. This study aimed to evaluate how information in the MSPE impacted LTI, with the hypothesis that changes in LTI would be relatively rare based on MSPE review alone. Methods: We conducted a prospective, observational study analyzing applications to three EM residency programs during the 2019-2020 match cycle. Reviewers assessed applications and rated the LTI on a five-point Likert scale where LTI was defined as follows: 1 = definitely no; 2 = probably no; 3 = unsure; 4 = probably yes; and 5 = definitely yes. The LTI was recorded before and after MSPE review. A change in LTI was considered meaningful when it changed the overall trajectory of the applicant’s likelihood to receive an invitation to interview. Results: We reviewed a total of 877 applications with the LTI changing ≥1 point on the Likert scale 160 (18.2%) times. The LTI was meaningfully impacted in a minority of applications – 48 total (5.5 %, p< 0.01) – with only 1 (0.11%) application changing from 1 or 2 (definitely/probably no) to 4 or 5 (probably/definitely yes) and 34 (3.8%) changing from 3 (unsure) to 4 or 5 (probably/definitely yes). Thirteen (1.5%) applications changed from 4 or 5 (probably/definitely yes) to 3 (unsure or probably/definitely no). Conclusion: Review of the MSPE resulted in a meaningful change in LTI in only 5.5% of applications. Given the time required for program leadership to review all parts of the variably formatted MSPEs, this finding supports a more efficient application review, where the PD’s focus is on succinct and objective aspects of the application, such as the Standardized Letter of Evaluation.