Real-World Performance of a Deep Learning Diabetic Retinopathy Algorithm

Abstract

Purpose : While prospective studies have investigated the accuracy of artificial intelligence (AI) for diabetic retinopathy (DR) and diabetic macular edema (DME), to date, little published data exists on real-world performance of these algorithms. Google and Verily partnered with ophthalmic care institutions in India to deploy the CE-marked AI for DR and DME. Since then, the algorithm (ARDA) has screened over 250,000 patients at 61 sites.

Methods : Our cross-sectional analysis evaluated the performance of ARDA in the post-deployment setting at Aravind Eye Hospital (AEH). Approximately 2% of fundus photographs were sampled from patients screened by ARDA between January 2019 and July 2022. Images were graded via adjudication by US ophthalmologists for DR and DME and ARDA’s output was compared against the adjudicated grades. Our primary analyses were the sensitivity and specificity of ARDA for severe+ DR (severe DR or proliferative DR (PDR)). Secondary analyses focused on sensitivity and specificity for sight-threatening DR (STDR) (DME or severe+ DR).

Results : Among the 4,874 eyes from 4,874 patients with adjudicated grades, mean age was 54.7 and 49.4% were female. 16.4% had any DR, 3.5% severe+ DR, 2.6% PDR, and 9.8% STDR. ARDA’s sensitivity and specificity for severe+ DR was 97.1% (95% CI 92.7-99.2) and 96.5% (95.9-97.1), respectively. The severe+ clinically important miss rate was 0.0% (e.g., some severe+ patients were interpreted as moderate and referred to clinic). ARDA’s sensitivity and specificity for STDR were 94.1% (91.2-96.3) and 96.5% (95.8-97.1), respectively.

Conclusions : In a post-deployment real-world setting, ARDA performed well, with sensitivity and specificity for severe+ DR exceeding 96% and catching 100% of severe+ patients for ophthalmology referral. To the best of our knowledge, this is the first large-scale post-marketing report of real-world performance for any medical device AI algorithm. Consistent with recommendations by regulatory bodies, we look forward to more post-marketing performance reports from the field.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.