Impact of PhORCAS references on overall application score for postgraduate year 1 pharmacy residency candidates

Abstract

Abstract Purpose The disparity between the number of applicants for postgraduate year 1 (PGY1) pharmacy residency positions and the number of available residency positions increases the need to optimize how applicants are evaluated. The purpose of the study described here was to evaluate the correlation of ratings of residency candidate characteristics by academic and professional references listed on residency applications with overall application score, applicant ranking, and the likelihood of candidates receiving an invitation to interview. Methods A multicenter, retrospective study was conducted to evaluate the correlation of reference writers’ ratings of 13 candidate characteristics and their overall recommendations with program-determined outcomes (eg, final application score, applicant ranking, and invitation to interview) through analysis of PGY1 applications submitted through the Pharmacy Online Residency Centralized Application System (PhORCAS) from 2015 through 2018. Keywords and themes within the open-ended section of letters of reference were also analyzed for correlation with overall application score. Results A total of 5,923 references listed on 1,867 applications to 4 PGY1 pharmacy residency programs processed by PhORCAS were included in the analysis. For the majority of applicant characteristic ratings (ie, 74% of 56,872 ratings overall), reference writers rated candidates as exceeding expectations, and applicants were “highly recommended” by these evaluators in 91% of cases. References’ average characteristic ratings and overall recommendations were poorly correlated with final application score (R2 = 0.12 [P < 0.0001] and R2 = 0.08 [P < 0.0001], respectively), final ranking (R2 = 0.02 [P < 0.0001] and R2 = 0.03 [P < 0.0001], respectively), and invitation to interview (R2 = 0.07 [P < 0.0001] and R2 = 0.04 [P < 0.0001], respectively). For the themes evaluated, references’ use of teaching words best correlated with normalized final application score, although the correlation was poor (R2 = 0.007, P = 0.0001). Conclusion Reference writers’ ratings of PGY1 residency candidate characteristics in PhORCAS are poorly correlated with application score, applicant ranking, and invitation to interview. The results of this study suggest that the existing PhORCAS standardized form for submitting references is of limited utility in its current state.