Automated Machine Learning Model for Fundus Photo Gradeability and Laterality: A Public ML Research Toolkit Sans-coding

Abstract

Purpose : To create a machine learning algorithm for fundus photo organization without coding. This model was shared with researchers as part of a broader research toolkit. It enables batch organization of large fundus photography datasets, and simultaneous exclusion of ungradable images in an automated fashion.

Methods : We utilized publicly available fundus photo datasets, and labeled them according to gradeability and laterality. Two retina fellows independently graded training and validation photos (n=3982), and an adjudication process was then performed with a third fellow deciding disagreements to set accurate ground truth labels. This was repeated with an external test dataset of 400 fundus photos. This test set comprised fundus photos from a different setting, country, and fundus camera. Google Cloud AutoML was then used to train a deep learning classification model through the online interface without the use of coding. An AUPRC was provided by the cloud platform, and confusion matrices were calculated. The resulting algorithm was exported as an edge model, and integrated into a framework for batch prediction. This was uploaded to GitHub for public use.

Results : Overall area under the precision-recall curve (AUPRC) was 0.96 for the internal testing dataset. Overall precision was 91.3%, and recall was 84.5%. When the threshold was adjusted to correspond with the intersect of precision and recall curves, the resulting per-label (precision, recall, [threshold]) were Left eye: (92.2%, 90.1%, [0.43]), Right eye (88.6%, 91.6%, [0.43], Ungradeable (77.4%, 81.3% [0.37]).

Conclusions : We demonstrate the use of automated machine learning sans-coding for the creation of a machine learning model to organize fundus photos. We published this for public use as part of a research toolkit. This may be used by researchers for organizing large datasets by eye laterality, and to exclude ungradable images testing their own deep learning models. Furthermore, we hope this encourages researchers without coding experience to design and publish their own automated deep learning models for the benefit of the research community.

This is a 2020 ARVO Annual Meeting abstract.