A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine

Abstract

Abstract Purpose In response to COVID-19, the American College of Physicians provided residents the option to complete the 2020 Internal Medicine In-Training Examination (IM-ITE) via in-person and remote proctoring. This study evaluated the extent to which scores obtained from both testing modalities were comparable. Method Data were analyzed from residents from all U.S.-based Accreditation Council for Graduate Medical Education–accredited IM residency programs and participating Canadian and international programs who completed the IM-ITE in 2020. The final sample contained 27,115 IM residents: 9,205 postgraduate year (PGY) 1, 9,332 PGY-2, and 8,578 PGY-3. Testing modality, gender, PGY, time spent on assessment, and native language were used to predict percent-correct scores in a multilevel regression model. This model included all main effects and all 2-way interactions between testing modality and each resident-level demographic variable, allowing those effects to be controlled for. Results Of 27,115 residents studied, 11,354 (42%) tested remotely and 15,761 (58%) in person. Across the parameters of interest (main effect of testing modality and 2-way interactions), the only statistically significant effects were the interaction effects between testing mode (interaction effect: −0.61; 95% confidence interval [CI], −1.01 to −0.21) and PGY (interaction effect: −0.54; 95% CI, −0.95 to −0.13) (P = .002). Differences between in-person and remote predicted scores were slightly larger for PGY-1 than for PGY-2 and PGY-3 residents, but the magnitude of these differences across residency training was well under one percentage point. Because these statistically significant effects were deemed educationally nonsignificant, the study concluded that performance did not substantively differ across in-person and remote examinees. Conclusions Residents taking the 2020 IM-ITE performed similarly across in-person and remote proctoring. This study provides evidence of score comparability across the 2 testing modalities and supports continued use of remote proctoring for the IM-ITE.