Measuring Competency: Improving the validity of your procedural performance assessments

Abstract

Objective: The objective of the study was to compare the use of ordinal scales and interval scales for capturing surgical competency information for general surgeons performing 3 complex trauma procedures. Background: Surgical performance assessment is typically captured using nonparametric data (eg, checklists) that do not support inferential analyses. Interval scales support parametric analyses that are essential for determining competency. We compared assessment outcomes for surgeons performing 3 complex trauma procedures using ordinal and interval scales. Methods: All participants were board-certified or eligible general surgeons. Each participant was assessed by an experienced trauma surgeon while performing 3 trauma procedures on cadavers. All assessors completed a rigorous assessment certification process. We calculated descriptive statistics to examine the differences between interval (parametric) and ordinal (nonparametric) outcomes. Results: Ordinal scales overestimated competence in up to 100% of the participants and did not identify specific performance gaps. Interval scales provided more granularity and identified specific capability gaps. Conclusions: Imprecise instrumentation conveys a false sense of competence and deprives surgeons of opportunities to close capability gaps. Measuring discrete procedural components with interval scales provides a more precise measurement of surgical competency.