Democratizing AI for DR: Automated Self-Training to Address Label Scarcity for Deep Learning in Diabetic Retinopathy Classification

Abstract

Purpose : To address the scarcity of high quality adjudicated labels for training supervised deep learning models. We used an automated sans-coding approach, developing a teacher model from a small high quality labeled dataset to subsequently assign diabetic retinopathy (DR) referral pseudolabels to a large unlabeled dataset. A student model was then trained from the combined dataset.

Methods : We utilized two publicly available fundus photo DR datasets. High quality adjudicated DR severity labels were applied to the first dataset of fundus photos (n=1744). Google Cloud Automated Machine Learning (AutoML) was next used to train a deep learning image classification model sans-coding, termed the teacher model for referable and nonreferable DR. The resulting algorithm was deployed in Google Cloud platform and made available for inference. The teacher model was used to generate DR referral predictions for an unlabeled public fundus photo dataset (n=58,689). The resulting image-label pairs were combined with the high quality teacher model’s training dataset, and a student deep learning model was trained.

Results : The teacher model area under the precision-recall curve (AUPRC), accuracy, sensitivity, specificity was 0.964, 92.0%, 73.9, 98.4 respectively, while the student model metrics improved to 0.976, 92.5%, 87.0%, 94.5% on the internal validation dataset. Teacher model external validation accuracy, sensitivity and specificity was 93.3%, 94.4%, 75.0%, while student model external validation metrics improved to 97.6%, 99.5%, 66.7% respectively.

Conclusions :
We demonstrate a sans-coding framework, which utilizes AutoML to address label scarcity for deep learning in DR. We show that self-training is an effective method to increase performance and decrease overfitting, which may simultaneously save time on expensive yet high-quality expert labeling. While the student model external validation dataset had 1 more false positive than the teacher model, clinically costly false negatives decreased by 10 as compared to the teacher model. As the tools for machine learning continue to be democratized, our methodology has potential to address the remaining disparity of expensive clinical labeling which bottlenecks small scale clinicians and researchers.

This is a 2021 ARVO Annual Meeting abstract.