Assistive artificial intelligence for improved clinical diagnosis of retinopathy of prematurity

Abstract

Purpose : Retinopathy of prematurity (ROP) is a leading cause of preventable childhood blindness. Current diagnostic methods rely on subjective clinical assessment of zone, stage, and plus disease, which can be variable between examiners. Previously, we developed a deep learning (DL) software as a medical device (SaMD), i-ROP DL, that outputs a vascular severity score (VSS) for objective quantification of the spectrum of plus disease. In this project, we evaluate the hypothesis that the use of a VSS may improve the diagnostic consistency of plus disease between experts.

Methods : Images from 150 eyes from 150 babies who underwent routine ROP screenings were selected—intentionally enriched for pre-plus and plus disease—from the Imaging and Informatics in ROP (i-ROP) consortium’s image-based dataset, collected January 2012–July 2020. Five retinal fundus images were obtained from each eye, and image-based reference standard diagnoses (RSD) for plus disease were assigned by 3 masked graders and the diagnosis from the ophthalmoscopic exam. Eleven ROP clinicians were tasked with assigning both a diagnosis of plus disease and a VSS grading to each examination. After one month, the same experts reassessed the images with an AI-derived VSS provided. We compared the area under the receiver operating characteristic curve (AUROC) for the clinician labeled VSS with and without AI against the RSD. Differences in AUROC were measured via Wilcoxon signed rank exact tests. Inter-rater agreement was assessed via weighted kappa.

Results : Clinician diagnosis of plus disease improved with the use of an AI-assigned VSS. AUROC (mean +/- standard deviation [SD]) for detection of pre-plus or worse disease improved from m 0.92 ± 0.08 to 0.96 ± 0.05 (p<0.001), and for plus disease improved from 0.94 ± 0.04 to 0.98 ± 0.2 (p=0.002). Finally, agreement between normal, pre-plus, and plus disease diagnoses improved from mean weighted Kappa ± SD 0.69 ± 0.14 to 0.81 ± 0.12 (p<0.001).

Conclusions : The use of an assistive SaMD for plus disease improved the mean accuracy and agreement for both pre-plus and plus disease among a group of experienced ROP clinicians. Future work is needed to demonstrate the effect on clinical outcomes.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.