Validation of Machine Learning Models and Comparison to a Simple Product Predictor (PP) to Identify Patients with Diabetes at Risk for Retinopathy

Abstract

Purpose : Diabetic Retinopathy (DR) causes preventable blindness. Previously, we found a DR prevalence of 21% in a Temple Primary Care Clinics (TPCC) screening program, suggesting that an annual diabetic eye exam for many patients is unnecessary. To optimize the frequency of DR screenings, we developed a simple predictor which worked as well as any of the 5 optimized machine learning models when resampling the training set. This study validates those findings using a new set of patients.

Methods : The development of the parent database and the optimization of the 5 machine learning algorithms (generalized linear model/logistic regression: GLM, support vector machine: SVM: recursive partitioning and regression trees: RPART, random forest: RF, and gradient boosted machine: GBM) were detailed in the derivation study. A subset of 540 different patients was used in this study. PP was defined as the product of HgA1c and years with diabetes (DMY). Prediction probabilities from all the models were determined for the test dataset and used to generate receiver operator characteristics curves (ROC). The area under each ROC (AUC)s for the 5 models were compared to the AUC from PP in a pairwise fashion using DeLong’s test. Dichotomous cutoff points for each of the 5 models and for PP were identified by maximizing Cohen's kappa. Confusion matrices were constructed against the known outcomes and the performance of the models, PP, and the no information predictor were compared for the normal retina and abnormal retina predictive groups using the chi square test. P values < .05 were considered significant.

Results : Table 1 shows AUC comparison of the 5 optimized models to PP. Only GBM had an AUC significantly greater than that of PP. Table 2 compares the predictions of normal retina by the naïve predictor, the 5 models and PP. RPART classified all patients as without retinopathy and was identical to the naive algorithm. All models and PP had significantly lower error rates than the naïve and RPART predictors; the error rates of PP and the remaining models were not statistically different.

Conclusions : This validation study suggests that PP is as good as complex machine learning models and has promise as a useful tool to identify diabetic patients at low risk for diabetic retinopathy.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

AUCs and p-values of Models

 

Ability of the Models to Identify Patients Without Retinopathy