Changing estimates of leadership ability before a programme: retrospective self-assessments and response-shift bias

Abstract

BackgroundMost evaluations of clinical leadership development programmes rely on self-assessments. Self-assessments are vulnerable to response-shift bias. Using retrospective then-tests may help to avoid this bias.In this study, we investigate whether post-programme then-tests (retrospective self-assessments) are more sensitive to change in clinical leadership development programme participants than traditional pre-programme pre-tests when paired with post-test self-assessments.Methods17 healthcare professionals participated in an 8-month single-centre multidisciplinary leadership development programme. Participants completed prospective pre-test, retrospective then-test and traditional post-test self-assessments using the Primary Colours Questionnaire (PCQ) and Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ). Pre–post pairs and then–post pairs were analysed for changes using Wilcoxon signed-rank tests and compared with a parallel multimethod evaluation organised by Kirkpatrick levels.ResultsA greater number of significant changes were detected using then-test pairs than pre-test pairs for both the PCQ (11 of 12 vs 4 of 12 items) and MLCFQ (7 of 7 vs 3 of 7 domains). The multimethods data showed positive outcomes at all Kirkpatrick levels.ConclusionsIn ideal circumstances, both pre-test and then-test evaluations should be conducted. We cautiously suggest that if only one post-programme evaluation can be conducted, then-tests may be appropriate means of detecting change.