Assessing the utility of a novel entrustment‐supervision assessment tool

Abstract

AbstractBackgroundWork‐based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment‐supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools.MethodsThe Ottawa Emergency Department Shift Observation Tool (O‐EDShOT) is a previously published WBA tool employing an entrustment‐supervision scale with strong validity evidence. This pre‐/post‐implementation study compares the performance of the O‐EDShOT with that of a traditional WBA tool using norm‐based anchors.All assessments completed in 12‐month periods before and after implementing the O‐EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor.ResultsA total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre‐ and post‐implementation phases respectively. The O‐EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O‐EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O‐EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O‐EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8.ConclusionThe O‐EDShOT outperformed a traditional norm‐referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment‐supervision scales generate more useful and reliable assessments in a variety of clinical settings.