Impact of training data diversity on the generality of automated diabetic retinopathy screening in fundus photographs

Abstract

Purpose : Several artificial intelligence (AI) systems are available for screening diabetic retinopathy (DR) in color fundus photographs (CFPs). Most of the existing systems have been developed using one large training dataset of conventional CFPs coming from one specific population. The aim of this study is to assess the impact of training data diversity (i.e. issued from different populations) on the generality of AI. Performance on unseen data acquired with a new device, namely the confocal DRSplus (iCare), is used to assess generality.

Methods : Two algorithms were developed for detecting moderate DR or worse in CFPs. A single-dataset (SD) algorithm was trained using one dataset: OPHDIAT (France, 176147 images). A multi-dataset (MD) algorithm was trained using three datasets: OPHDIAT, Kaggle (U.S, 31648 images) and DDR (China, 8763 images). Optionally, the MD algorithm was fine-tuned using 1578 DRSplus images (fine-tuned MD). All the algorithms were evaluated on 378 independent DRSplus test images.

Results : The area under the curve (AUC) for detecting at least moderate DR in DRSplus test images was 0.9856 for the SD algorithm and 0.9908 for the MD algorithm. The performance of the MD algorithm was poorer after fine-tuning on DRSplus training images (AUC = 0.9812). For a sensitivity of 0.9412, the specificity was 0.9244 for the SD algorithm, 0.9535 for the MD algorithm and 0.9012 for the fine-tuned MD algorithm. For a specificity of 0.9767, the sensitivity was 0.7647 for the SD algorithm, 0.8824 for the MD algorithm and 0.6471 for the fine-tuned MD algorithm.

Conclusions : Despite the visual difference existing between conventional and confocal-based CFPs, the algorithms trained on conventional CFPs showed good performance for detecting moderate DR or worse in confocal-based DRSplus CFPs. The MD algorithm, trained on different populations, showed better performance than the SD algorithm. This study suggests that the training data diversity improves the generality of AI. The fact that fine-tuning on a few DRSplus images does not improve performance highlights this generality.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Receiver operating characteristic curves for detecting moderate DR or worse on test DRSplus images for the SD algorithm, the MD algorithm and the fine-tuned MD algorithm. The points A, B, C and D, E, F denote the performances of those algorithms at a 0.9767 specificity and 0.9412 sensitivity cut-off respectively.