Variability in plus disease diagnosis using single and serial retinal images

Abstract

Purpose : There has been demonstrated variability in plus disease diagnosis amongst ROP experts. There are subjective differences in the diagnostic cutoffs for pre-plus and plus disease. Features such as training, field of view, magnification, and tempo have been proposed as explanations. The purpose of this study is to assess changes in ROP diagnosis in single and serial retinal images.

Methods : 7 graders with expertise in ROP grading independently reviewed both single and 3 consecutive serial retinal images from 15 ROP cases on a secure web-based platform. Severity was assigned as plus, pre-plus, or none. A secondary analysis was performed using the previously validated i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1-9, with 9 being most severe disease. This score has been previously demonstrated to correlate with expert diagnosis and the ICROP. Mean plus disease severity was calculated by averaging 14 labels per image in both serial and single images.

Results : Assessment of serial retinal images changed the grading severity for >50% of the graders, though there was wide variability. Cohen’s kappa ranged from 0.29 to 1.0. Changes in grading of serial retinal images was noted more commonly in cases of pre-plus disease, where the mean severity showed a borderline significant increase (p=.08). (Figure 1) The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease, and overall agreement with the mode class (p=0.001) (Figure 2A). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient 0.89). (Figure 2B). The VSS also demonstrated agreement with disease progression across serial images which progressed to pre-plus and plus disease. (Figure 2C).

Conclusions : Clinicians demonstrated diagnostic variability with both single and serial images. However, the use of serial retinal images caused a change from pre-plus to plus disease, which represents a change in management. More aggressive graders tended to be influenced by serial images to increase the severity of their grading. The use of deep learning as a quantitative assessment of plus disease can standardize diagnosis and treatment.

This is a 2021 ARVO Annual Meeting abstract.

 

The color and numerical scal correlates to disease severity, ranging from most mild (green, 1) to most severe (red, 9).

 

Vascular severity score in serial retinal images (represented at three time points, T-2, T-1, and T-0)