Generalizable AI-based glaucoma prediction via a stable model selection method

Abstract

Purpose : Deep Learning (DL) models have a high tendency for overfitting in small-labeled data regimes. Given the limited amounts of labeled data in the Ophthalmology domain, we propose a novel DL model selection method to avoid overfitting in small data regimes for the glaucoma classification task.

Methods : We used 1340 fundus images from the Illinois Eye and Ear Infirmary with 683 glaucoma and 657 non-glaucoma patients. Table 2 summarizes the number of images in the train, validation, and test sets. The baseline classification (BCL) method selects the best model based on the max accuracy on the validation set. In contrast, we propose a method that selects the best model not only based on high accuracy but also low variability to improve the model’s generalizability on the unseen test data. Specifically, our method selects the best model based on both average (μ) and standard deviation (SD) of validation accuracy over Ν consecutive epochs (e.g., Ν = 10) across Κ hyper-parameter searches (e.g., Κ = 30). We benchmarked BCL against our proposed method to compare the generalizability of the methods on the unseen test data for classifying glaucoma based on fundus images.

Results : As shown in Table 1, our proposed method outperforms BCL in classifying glaucoma using almost any validation set size. Specifically, our method outperforms BCL by improving the test accuracy by 17%, 11%, 2%, and 9% for validation sizes of 2, 10, 30, and 300, respectively. BCL outperforms our method by improving the test accuracy by 1% when the validation size equals to 100. Further, our proposed method, in contrast to BCL, predicts glaucoma with high accuracies of 79% and 85% when solely there are 2 and 10 images in the validation set, respectively.

Conclusions : We showed BCL is not robust against overfitting in a small data regime, and it cannot predict glaucoma accurately when the train and validation data are limited. However, our proposed method improves the performance of the baseline for predicting glaucoma, and it generalizes to the unseen test data robustly across various validation set sizes. Therefore, we showed the importance of minimizing the variance of the model while maximizing the accuracy to select a high-performance, robust, and generalizable model for glaucoma prediction.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Table 1. Data breakdown.

 

Table 2. Test set performance is shown as accuracy % (AUC %). Bold values show the max accuracy.