Abstract
Purpose :
Deep convolutional neural networks (CNN) have successfully automated retinal image segmentation. Their performance, however, depends heavily on the properties of the dataset used in their development. In previous work, our group has shown that models trained on diverse datasets perform poorer on single-domain tasks but better on domain adaptation, compared to models trained on less heterogenous datasets, which showed the opposite pattern. Here we examine the performance of a retinal image segmentation model developed using the Indian Diabetic Retinopathy Image Dataset (IDRiD), when tested on a subset of EYEPACS, a dataset collected from a primarily Latin American and Caucasian population.
Methods :
We trained a UNet3 model using the IDRiD dataset to segment hard exudates in retinal fundus images. We randomly partitioned the IDRiD dataset (N=54) into Training (80%), Validation (10%), and Test (10%) sets. We also created a secondary validation set (gold standard) for hard exudate counts for 100 retinal images (50 Females, 84% Latin American) from the EYEPACS dataset where hard exudate counts were generated by an ophthalmologist (Dr. Ozturan) and tested the IDRID-trained UNet3 model on the EYEPACS distribution to assess domain adaptation performance.
Results :
The Test AUC for the UNet3 model was 0.78. On the EYEPACS Validation set, the model's exudate counts (M=7.54 SD=17.41) were similar to the ophthalmologist counts (M= 7.66, SD=21.78, p>0.2). However, a close inspection of the segmentation outputs revealed that the model missed at least one lesion on 26% of all images, averaging 9.38 missed hard exudates. It also incorrectly labeled false lesions on 52% of all images, with an average of 4.35 false labels, misidentifying features like peripapillary atrophy, soft exudates, optic disc vessels, RNFL light reflexes, vessel light reflex, RPE changes, retinal atrophy, as hard exudates.
Conclusions :
Our findings underscore the significance of dataset diversity for the generalization performance of CNNs in healthcare. While the UNet3 model showed reasonable accuracy, its inconsistencies in lesion detection highlight the limitations of using a CNN trained on a homogeneous dataset for diverse image distributions. These findings emphasize the necessity of matching the development set with the target population's diversity in AI applications, particularly in healthcare settings.
This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.