Interrater reliability of photographic assessment of thyroid eye disease using the VISA classification

Abstract

Abstract Purpose To determine the interrater reliability (IRR) of thyroid eye disease (TED) photographic assessment using the VISA classification. To assess whether a VISA grading atlas improves ophthalmology trainees’ performance in photographic assessment of TED. Methods A prospective, partially randomized, international study conducted from September 2021 to May 2022. Online study invitation was emailed to a volunteer sample group of 68 ophthalmology college accredited consultants and trainees, and 6 were excluded from the study. Participants were asked to score 10 patient photographs of TED using only the inflammation and motility restriction components of the VISA classification. IRR was compared between groups of practitioners by their level of experience. A clinical activity grading atlas was randomized to 50% of the ophthalmology trainees. Results Overall rater ICC was 0.96 for inflammation and 0.99 for motility restriction. No statistically significant difference in IRR between rater groups was identified. Trainees with a grading atlas had the highest IRR for inflammation (ICC = 0.95). Each subcomponent of the inflammation and motility restriction components of VISA classification had an ICC considered good to excellent. The mean overall rater score was 4.6/9 for inflammation and 3.5/12 for motility restriction. For motility restriction there was a reduced mean score variance among all raters when scoring photographs with more severe motility restriction. Conclusion IRR using the inflammation and motility restriction components of the VISA classification was excellent. A VISA grading atlas improved trainee performance in grading inflammation.