Ethnic Diversity of Retinal Images Used to Train Artificial Intelligence Models Improves Diagnostic Accuracy to Detect Diabetic Retinopathy

Abstract

Purpose : To compare the performance of automated machine learning (autoML) models to detect diabetic retinopathy (DR) using a diverse versus non-diverse training dataset of retinal images.

Methods : A previously validated AutoML (Google Cloud) model to detect DR was generated based on 5-field handheld retinal images (N=16,681) from a Filipino (Malay/Asian, FIL) population. Additional images using the same camera from an Egyptian, Finnish, and Tanzanian (EGY, FIN, TANZ) cohort were tested against this model (N=641,767,215). The baseline algorithm trained with Filipino eyes was re-trained with additional eyes from different ethnicities (Arab, White, Black). The performance of the augmented algorithm was retested on a holdout set from the same 3 ethnicities. Image labeling was performed at a reading center using the International DR and diabetic macular edema (DME) scale. Referable DR (refDR) was defined as moderate nonproliferative DR or worse, or DME.

Results : RefDR per ethnicity (FIL, EGY, FIN, TZA, %): 17.3/87.8/18.5/27.9. Baseline performance of the model trained on Filipino eyes to detect refDR per ethnicity: 0.995/0.838/0.931/0.856 [area under the precision-recall (AUPRC), Fig. 1]. For all non-Asian eyes combined (N=1,623), the baseline Filipino model had an AUPRC of 0.901. After augmenting the baseline Filipino model with non-Asian eyes, AUPRC for detection of refDR in the whole cohort increased to 0.957. Number of images for validation of the model per ethnicity: 225/164/192/49. Sensitivity/specificity (SN/SP) of the baseline Filipino model tested against the FIL, EGY, FIN, TZA images: 0.96/0.98, 0.77/0.90, 0.44/1.00, 0.67/1.00. For the augmented model, SN/SP for each ethnicity improved to 0.96/1.0, 0.93/0.84, 0.62/0.99, 0.92/0.97. The augmented versus baseline model significantly improved (p=0.001) the prediction of refDR in the combined non-Asian cohorts (EGY, FIN, TAZ).

Conclusions : The performance of the initial autoML model to detect refDR developed from Filipino eyes was lower when tested on non-Asian datasets. Augmenting the model with non-Asian eyes potentially improves performance in non-Asian test sets without a decrease in performance in Asian eyes. Our data stresses the importance of diverse datasets, and the need to test and optimize AI models in the patient populations they will be used.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.