Orthopedic in-training examination question metrics and resident test performance

Abstract

First administered in November 1963, the orthopedic in-training examination (OITE) is now distributed to more than 4000 residents in over 20 countries and has become important for evaluation of resident fund of knowledge. Several studies have assessed the effect of didactic programs on resident performance, but only recently has it become possible to assess detailed testtaking metrics such as time spent per question. Here, we report the first assessment of resident OITE performance utilizing this full electronic dataset from two large academic institutions. Full 2015 OITE score reports for all orthopedic surgery residents at two institutions were anonymized and compiled. For every question answered by each resident, the resident year, question content or domain, question result (correct or incorrect), and answer speed were recorded. Data were then analyzed to determine whether resident year, result, or domain affected answer speed and whether performance in each subspecialty domain varied based on resident year in training. Data was available for 46 residents and 12,650 questions. Mean answer speed for questions answered correctly, 54.0±48.1 s, was significantly faster than for questions answered incorrectly, 72.2±61.2 s (P<0.00001). When considering both correct and incorrect answers, PGY-1s were slower than all other years (P<0.02). Residents spent a mean of nearly 80 seconds on foot and ankle and shoulder and elbow questions, compared to only 40 seconds on basic science questions (P<0.05). In education, faster answer speed for questions is often considered a sign of mastery of the material and more confidence in the answer. Though faster answer speed was strongly associated with correct answers, this study demonstrates that answer speed is not reliably associated with resident year. While answer speed varies between domains, it is likely that the majority of this variation is due to question type as opposed to confidence. Nevertheless, it is possible that in domains with more tiered experience such as shoulder, answer speed correlates strongly with resident year and percentage correct.