Iterative Machine Learning: A Test Case for the Detection of Disc Hemorrhage

Abstract

Purpose : Google AutoML, an online machine learning platform, has demonstrated comparable performance to hard-coded deep learning (DL) algorithms for the identification of pathology in fundus photos. Here we explore whether a small labeled training set can be leveraged to train AutoML to detect disc hemorrhages (DHs) in a large unlabeled test set using an iterative approach to prepare data with reduced time burden.

Methods : A retrospective analysis of fundus photographs of different resolutions and sources centered on the optic disc obtained from the Ocular Hypertension Treatment Study, New York Eye and Ear Infirmary of Mount Sinai, Massachusetts Eye and Ear, Thessaloniki Eye Study, and GONE datasets were included. Ground truth was established by the consensus of grades from two masked glaucoma specialists. We trained AutoML to distinguish DH positive and DH negative images. Predictions for a held-out validation set were obtained in batches. Positive predictions over a confidence threshold of 0.9 confirmed by a glaucoma specialist were added to the positive dataset to retrain the model. This process was repeated for the entire held out dataset. Area under the receiver operator curve (AuROC) for the AutoML model and a custom model based on the Inception v3 architecture were calculated.

Results : Baseline DH identification model was trained using 391 case and 990 control images. A held-out validation set of 80,000 images was partitioned into 8 batches. Classifications by the baseline model of the 1st batch yielded 2521 positives (out of 13530) with 51 true positives on review (Positive Predictive Value (PPV) = 0.020). AutoML Model v2 included these 51 positive images, testing with a 2^nd batch produced 1182 positive classifications (of 10912) with 27 positives on manual review (PPV = 0.023). Repeating this process for 8 iterations identified 275 DH positive images. The PPV of the final AutoML Model was 0.400 (Fig 1). The AutoML and Inception models achieved AuROCs of 0.79 and 0.64, respectively, in the final imageset. False positive examples depict challenges in learning the DH phenotype (Fig 2).

Conclusions : DHs are rare events in glaucoma patients and somewhat resistant to machine learning. Iterating on a baseline model using a large held out dataset led to a 20-fold increase in PPV.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

 

Shunt vessels or hemorrhage away from ONH were common culprits for FP classifications and were included as negative cases for subsequent training.