Not so simple: evaluating consequences validity evidence for a workplace-based assessment in surgery

Abstract

Abstract Purpose Workplace-based assessments (WBAs) of trainee operative skills are widely used in surgical education as formative assessments to facilitate feedback for learning, but the evidence to support this purpose is mixed. Further evaluation of the consequences of assessment use and score interpretation is needed to understand if there is alignment between the intended and actual impacts of assessment. This study examines consequences validity evidence for an operative WBA, exploring whether WBA use is consistent with the goals of formative assessment for learning. Methods Eight residents and 9 faculty within the Department of Otolaryngology—Head and Neck Surgery at a tertiary institution completed semi-structured interviews after participating in a pilot of a surgical WBA, the System for Improving and Measuring Procedural Learning in the OR (SIMPL OR). Residents received feedback from attendings via both scores (performance and autonomy ratings) and recorded dictations. Interview questions explored faculty and resident perceptions of feedback behaviors and perceived impacts on their teaching or learning practices. Three researchers analyzed transcripts using directed qualitative content analysis to generate themes and evaluated how the perceived impacts aligned with formative purposes for assessment and score use. Results Both faculty and residents identified intended impacts of formative assessment, including (1) greater emphasis on feedback, (2) support for a postoperative feedback routine, and (3) facilitation of case-specific reflection. Residents also used score and verbal feedback for (1) calibrating case perceptions and (2) benchmarking performance to an external standard. The recorded dictations supported feedback by (1) providing context for ratings, (2) facilitating review of dictated feedback, and (3) prompting faculty for deliberate feedback. Unintended impacts included: (1) emotional discomfort during the assessment process, (2) increased feedback frequency but not diversity or quality, (3) inadequate support for feedback conversations, and (4) limited next steps for teaching or learning. Assessment usage declined over the pilot period. Conclusions The validity evidence gathered in this study suggests an operative WBA can be used for formative purposes to improve perceptions of feedback, but unintended consequences and implementation challenges limited ultimate impacts on teaching and learning. User perspectives can add important elements to consequences validity evidence and should be further evaluated in different implementation settings to better understand how WBAs can achieve their formative goals.