Inter-rater variability in glaucoma diagnosis using optic nerve head photographs

Abstract

Purpose : Characterizing optic nerve head (ONH) photographs is fundamental to the care of patients in the glaucoma clinic. However, diagnosis of glaucoma is known to be variable. The goal of this study was to quantify inter-rater variability in disease classification, to examine potential biases between examiners, and to investigate if differences in diagnoses arise from different thresholds in classifying disease states. In the second part of the study, we examine if the ranking of disease severity instead of disease level classification improves concordance between clinicians.

Methods : 158 images from the publicly available RIM-one dataset were selected for analysis for the disease classification into one of 4 classes (normal, mild, moderate, and severe). 50 images were selected for the image ranking phase where each image was compared to all other images. Based on the commonly used Elo algorithm, the gradings from the pairwise comparisons were used to create a ranked order of images. We also create a consensus rank based on gradings from all raters.

Results : There was substantial variability between clinicians in their classification of disease severity based on the ONH photos. Pairwise kappas between raters ranged from 0.22 to 0.57. Agreement with the original ground truth provided by the data generators of the RIM-one dataset varied from 0.25 to 0.59 suggesting that there is relatively modest agreement between clinicians when assessing disease severity based on ONH photos.
However, the agreement between clinicians in ranking the images was substantially higher with correlation coefficients between each clinician and the consensus ranking varying between 0.72 and 0.95.

Conclusions : We demonstrate that glaucoma disease severity assessment based on ONH photos is highly subjective with clinicians utilizing different cut-points. We also show that even though there is substantial difference in the disease classification, there is much higher agreement in the ordering of disease severity, especially among two senior clinicians . We propose that using pairwise comparisons to establish a rank ordering of images might provide a more quantitative disease severity assessment in glaucoma.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

A. Pairwise kappas between clinicians
B. Color coding of disease severity, rows are raters, columns are images ordered by average rating.

 

The ranking of severity shows good agreement between the senior clinicians and consensus ranking