Binary versus 5-Point Likert Scale in Assessing Otolaryngology Trainees in Endoscopic Sinus Surgery

Abstract

Objectives: (1) Examine the discriminant validity of 2 validated assessment tools in endoscopic sinus surgery (ESS) training. (2) Compare the binary versus the 5-point Likert-scale tools in ESS assessment. Methods: A cross-sectional study was conducted from February-August 2013. All otolaryngology trainees in the North-London training program were assessed while performing ESS on sheep heads, which previously showed face and content validity. Performance was rated by 2 blinded assessors using 2 validated tools: The Inter-Collegiate-Surgical-Curriculum-Project tool, which utilizes a binary system and is used throughout the UK surgical training system, and the John Hopkins 5-Point Likert scale ESS tool. The tools’ construct validities were tested by comparing performance of experts and novices. The tools were also tested by correlating task-specific and global skills ratings with overall performance level. Results: The binary tool showed higher inter-rater reliability than the Likert scale, both in task-specific (Kappa: 0.89 versus 0.62) and global skills (Kappa: 0.79 versus 0.68) rating. Both tools discriminated between different levels of expertise in global and task-specific skills (Kruskal-Wallis: P < .001 for all). But pairwise comparison showed that Likert scale tool was a better discriminator between novices and trainees, globally ( P = .016 versus P < .271) and task-specifically ( P = .049 versus P = .449). Likert scale tool also correlated better with overall performance globally ( r = 0.903 versus 0.741) and task-specifically ( r = 0.833 versus 0.802). Conclusions: While the binary assessment tool may show higher inter-rater reliability and is easier to complete, the Likert scale, when each level is well defined, showed good agreement and better construct validity, which can be useful when monitoring progress in training.