Evaluation of equity in performance of Artificial Intelligence for diabetic retinopathy (DR) detection

Abstract

Purpose : Given rising prevalence of diabetes, costs and workload associated with screening for diabetic eye disease is growing. Automated retinal image analysis systems (ARIAS) for DR offer a solution by partially replacing human grading of images. We examined performance equity in an independent head-to-head evaluation of ARIAS in a real-life setting.

Methods : 8 out of 22 potentially eligible CE Class IIa systems for DR detection from retinal images agreed to participate. From 202,886 screening encounters at North East London Diabetic Eye Screening Programme (1^st January 2021-31^st December 2022) we curated a database of 1.2 million images and sociodemographic/grading data. Images were manually graded by up to 3 graders following a standard national protocol and final human grade in worst eye was used as the reference standard. ARIAS vendors received sample data from 1000 expired encounters (6000 images), national grading guidance and image capture protocols to confirm ARIAS outputs could be replicated in the research environment. Sensitivity and false-positive rates (95% confidence intervals) were determined by age quartiles, self-reported ethnicity (39% South Asians, 32% white and 17% of black) and sex according to level of DR. ARIAS outputs for test-positive/technical failure versus test-negative were compared with the standard. Vendor algorithms did not have access to human grading data.

Results : Performance was stable across population subgroups of age, sex, ethnicity for moderate-to-severe non-proliferative DR and for proliferative DR, with sensitivities across vendors ranging from 95.6% (94.9%,96.2%) to 99.8% (99.6%,99.95%) and 96.8% (95.9%,97.5%) to 99.4% (98.9%,99.75%) respectively. Sensitivities for mild-to-moderate non-proliferative DR with referable maculopathy ranged from 73.9% to 98.3% with heterogeneity in performance by ethnicity/age groups for some vendors. Similarly false positive rates for no observable DR ranged from 4.3% to 61.4% and varied by population subgroups.

Conclusions : Algorithms demonstrated safe levels of sensitivity for medium/high-risk DR in a real-world screening service which were equitable by subgroups of age, sex and ethnicity. For lower levels of DR or no DR, test positive rates varied by population subgroups. ARIAS can provide clinically equivalent, rapid detection DR for triaging no-DR/low vs. medium/high risk DR cases.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.