Interobserver Reliability of Rating Color Slides to Measure Change in Graves’ Ophthalmopathy

Abstract

Abstract: : Purpose: The use of color slides has been reported to be a reliable method for assessing treatment response in Graves’ ophthalmopathy, and its use as an outcome measure in clinical trials has been advocated (1). We report interobserver reliability among six raters using color slides to assess change in Graves’ ophthalmopathy patients. Methods: Color slides were taken before and 6 months after randomly assigned, unilateral external beam radiotherapy in 42 Graves’ ophthalmopathy subjects participating in the Orbital Radiation in Graves’ Ophthalmopathy randomized trial (2). All subjects had mild or moderate bilateral Graves’ ophthalmopathy. The baseline and after-treatment slides for each subject were identified and projected side-by-side in a single session for six experienced Graves’ ophthalmopathy observers (three endocrinologists, two ophthalmologists, and one radiotherapist). The observers were instructed to specify whether the right eye, left eye, or neither eye had the better outcome. The kappa statistic was calculated as a measure of interobserver reliability. Results: The six observers showed varying degrees of preference for the three classification categories. One observer rated 27 of the 42 photograph pairs (64%) as favoring neither eye, while another observer used the neither eye category for only 11 of the photograph pairs (26%). The kappa statistic was 0.088 for the two endocrinologists, 0.22 for the three ophthalmologists, and 0.21 for all six observers. Conclusions: Among six experienced Graves’ ophthalmopathy observers, agreement in the assessment of treatment benefit using color slides was only modestly elevated above what would be expected by chance alone. These data suggest that further efforts are needed to develop valid, reliable outcomes for the assessment of change in Graves’ ophthalmopathy. References: 1. Gerding MN, Prummel MF, Kalmann R, et al. The use of colour slides in the assessment of changes in soft-tissue involvement in Graves’ ophthalmopathy. J Endocrinol Invest 1998; 21:459-62. 2. Gorman CA, Garrity JA, Fatourechi V, et al. A prospective, randomized, double-blind, placebo-controlled study of orbital radiotherapy for Graves’ ophthalmopathy. Ophthalmology 2001; 108:1523-34.

Keywords: clinical (human) or epidemiologic studies: sys • orbit