Using Multi-Layer Perceptron Driven Diagnosis to Compare Biomarkers for Primary Open Angle Glaucoma

Abstract

Purpose: To use neural network machine learning (ML) models to identify the most relevant ocular biomarkers for the diagnosis of primary open-angle glaucoma (POAG).

Methods: Neural network models, also known as multi-layer perceptrons (MLPs), were trained on a prospectively collected observational dataset comprised of 93 glaucoma patients confirmed by a glaucoma specialist and 113 control subjects. The base model used only intraocular pressure, blood pressure, heart rate, and visual field (VF) parameters to diagnose glaucoma. The following models were given the base parameters in addition to one of the following biomarkers: structural features (optic nerve parameters, retinal nerve fiber layer [RNFL], ganglion cell complex [GCC] and macular thickness), choroidal thickness, and RNFL and GCC thickness only, by optical coherence tomography (OCT); and vascular features by OCT angiography (OCTA).

Results: MLPs of three different structures were evaluated with tenfold cross validation. The testing area under the receiver operating characteristic curve (AUC) of the models were compared with independent samples t-tests. The vascular and structural models both had significantly higher accuracies than the base model, with the hemodynamic AUC (0.819) insignificantly outperforming the structural set AUC (0.816). The GCC + RNFL model and the model containing all structural and vascular features were also significantly more accurate than the base model.

Conclusions: Neural network models indicate that OCTA optic nerve head vascular biomarkers are equally useful for ML diagnosis of POAG when compared to OCT structural biomarker features alone.

The mechanisms and specific risk factors causing primary open-angle glaucoma (POAG) remain largely unidentified despite the disease being one of the leading causes of irreversible blindness worldwide.^1,2 Historically, intraocular pressure (IOP) has been characterized as a leading risk factor and therapeutic target, yet up to half of the glaucomatous patients in the United States show disease progression despite well-maintained IOP.³ Research has also implicated advancing age, ocular vascular abnormalities,^4–6 and genetics⁷ with the onset and progression of POAG. Glaucoma affects individuals differently, suggesting subgroups of patients exist where certain risk factors combine or are more relevant than in others, and vice versa.⁸ 

State of the art glaucoma diagnosis is done through assessment of structural damage to the retina and optic nerve head (ONH) and visual field (VF) defects. Although the definition of POAG often lacks consensus, damage to the ONH, reduced retinal nerve fiber layer (RNFL) thickness, and a decrease in the thickness of the macular ganglion cell complex (GCC) are indicative of POAG. Diagnostically, assessment of RNFL thickness and visual inspection of the ONH (i.e., for cupping measured by the cup-to-disk ratio) accompanied by VF defects are considered the standard of care for POAG.⁹ Because of the heterogeneity of POAG and high variability in individual susceptibility, the biomarkers that best predict glaucoma onset and progression for a given individual or group have not been established. This complex data challenge represents a significant barrier to providing precision care for POAG patients. 

Alongside elevated IOP, a variety of reported vascular abnormalities have been associated with both the onset and progression of POAG.^4–6,10 Multiple imaging techniques have identified microvascular defects in the retina, slower choroidal flow, and vasospasm in the retrobulbar blood vessels in POAG patients.⁵ Optical coherence tomography angiography (OCTA) is a more widely available noninvasive imaging modality that has been used over the past decade to characterize blood vessel densities (VD) and circulation in the retina and ONH in patients with POAG.^6,11 Yarmohammadi et al.¹² compared POAG diagnosis using OCT-derived RNFL thickness and OCTA-derived VD finding that glaucomatous patients, glaucoma suspect, and controls had significantly different VD and RNFL thickness with both values decreased for glaucoma patients compared to controls. Although OCTA-assessed VD loss has been linked to both POAG disease and functional vision loss,^6,12 they are not included in most diagnostics or the current standard of care. The complexity of incorporating OCTA VD and other hemodynamic biomarkers together with IOP, and ONH and RNFL structure in a statistical model limits their translation and potential impact to inform on POAG. 

In the last decade there have been numerous attempts to augment glaucoma diagnosis using artificial intelligence (AI).¹³ AI techniques have compared multiple kinds of machine learning (ML) algorithms,¹⁴ and either feed retinal images directly^15–20 or as a set of features based on algorithmic feature extraction.^21–23 Often these techniques compare diagnostic accuracy between different ML algorithms¹³ or against human doctors.²⁴ Medeiros et al.¹⁹ and Burgansky-Eliash et al.²⁰ both used OCT images for ML-aided glaucoma investigation. Medeiros et al.¹⁹ used convolutional deep learning (DL) algorithms to quantify glaucomatous damage on OCT images and compared with human graders. Burgansky-Eliash et al.²⁰ found that support vector machines (SVMs) reached the highest diagnostic area under the receiver operator characteristic curve (AUC) with OCT-derived parameters and reported an AUC of 0.981. This literature provides support that computer-aided diagnosis (CAD) for glaucoma diagnosis using OCT images provides promising results.²⁰ Another recent study by Mariottoni et al.²¹ used DL to assess more than 14,000 spectral-domain (SD) OCT-derived RNFL thickness measurements to develop a model that can determine the probability of structural progression in 816 glaucomatous eyes of 462 individuals. After analysis their DL model boasted an AUC of 0.938 (95% confidence interval [CI], 0.921-0.955), not only outperforming conventional trend-based analyses of progression but also indicating the most likely areas of the ONH to experience progression. These results indicate that DL in conjunction with OCT may prove to be a useful tool for clinicians when assessing the probability and location of POAG progression. 

Based on the literature results neural network-based algorithms (e.g., deep learning, convolutional neural networks, multilayer perceptrons (MLPs) and SVMs have been empirically determined to be the most successful to date for glaucoma diagnostics.^13,14,20,21 MLPs allow for the ability to change the model complexity easily, in an effort to avoid overfitting when using feature vectors of various sizes. SVMs also require a choice of kernel function for transforming the data, and choice of kernel function can greatly change the results of the model. MLPs do not use a kernel, instead MLP parameters include their structure, initialization strategy, and activation function. Thus, multiple structures of MLPs were used in this study to compare performance across feature sets with inputs of different dimensionality. Further, the accuracy of these models is similar if not slightly better than the performance of human graders on the same task.²³ Despite these results, the decision-making process used by neural networks are not interpretable, a problem often referred to as “black box models” and can vary across studies, which currently limits the clinical application of these systems. 

Diagnosis of POAG is complex and lacks a gold standard, especially in the earliest stages of the disease. To identify the clinical biomarkers that may be the most salient, a series of ML models should be trained to assess AI or CAD accuracy. The reasoning behind this approach is that a model that diagnoses glaucoma with only one feature (i.e., IOP) is very likely to do worse than a model that includes IOP alongside average RNFL thickness. RNFL thickness is commonly used for glaucoma diagnosis in practice, providing more glaucoma-related information than the feature of IOP alone. Thus the feature set that produces the highest diagnostic accuracy likely contains the most clinically relevant information that can be exploited by a CAD system. This method reflects that in Burgansky-Eliash et al.²⁰ where the top eight training features that correlated with the ML model accuracy were identified, and a CAD system was trained using only these. Conversely, our approach segments the biomarkers by the kind of information (e.g., structural or hemodynamic, etc.), and both methods aim to identify the top-performing biomarkers for AI diagnosis. 

The purpose of the present study was to compare the diagnostic accuracy of MLP algorithms for diagnosing POAG given different features calculated from clinical ophthalmic data and OCTA biomarkers. We compare different feature sets on the same type of ML model to examine how useful a given feature set may be for diagnosis, relative to the other feature sets. The results provide information on the usefulness of OCTA vascular features for AI-assisted diagnosis, as well as a comparison of sets of features that best inform for POAG diagnosis. 

Material and Methods

Participants and Examination

This study considers a prospectively collected observational dataset comprised of 144 POAG patient eyes and 149 non-glaucomatous controls without eye disease. Enrolled participants were required to be at least 21 years of age, signed an informed consent, and all methods adhered to the tenets of the Declaration of Helsinki and the policies and regulations of the Institutional Review Board at the Icahn School of Medicine at Mount Sinai. A certified glaucoma specialist determined POAG status considering the presence of an open angle and classical glaucomatous structural damage at the level of the ONH and RNFL thickness, and functional VF defects, regardless the level of IOP. The non-glaucomatous control eyes consisted of participants with both eyes being free of any eye diseases including POAG. Exclusion criteria included the following: refractive error >9 Diopters and <−9 D in spherical equivalent; evidence of exfoliation or pigment dispersion; eye disease other than glaucoma; use of ocular medications (other than IOP-lowering medications for glaucoma or eye lubricants for dry eye); neurological disease; psychosis or other diseases that could prevent reliable eye examinations; severe, as well as unstable or uncontrolled cardiovascular, renal, or pulmonary disease. 

All study subjects underwent a complete eye examination by a glaucoma specialist including history, visual acuity testing, refraction, slit-lamp biomicroscopy, gonioscopy, ultrasonic pachymetry, and dilated ophthalmoscopy. Heart rate (HR), systolic blood pressure (SBP), and diastolic blood pressure (DBP) were assessed with automated ambulatory cuff after five minutes rest. IOP was assessed with Goldmann applanation tonometry. All study subjects also had stereo disc photography (Visucam Pro NM; Carl Zeiss Meditec, Inc., Jena, Germany) and VF testing (Swedish Interactive Threshold Algorithm 24–2 test of the Humphrey VF analyzer 750i; Carl Zeiss Meditec, Inc.). 

OCTA imaging was assessed in all study eyes (RTVue XR, Version 2018.1.1.63; Optovue Inc, Fremont, CA, USA) providing three-dimensional visualization of the retinal microvasculature and OCT-derived structural parameters at the level of the ONH. AngioAnalytics licensed upgrade was used to provide separate VD analysis computed as percentage of area occupied by OCTA detected vasculature at the level of the macula and ONH. VD was assessed for the small vessels (SV, i.e., with large vessel masking: large vessel mask had a threshold of ≥3 pixels [approximately ≥33 µm]) and for all vessels with the 4.5 mm HD Angio Disc scan in the radial peripapillary capillary slab ranging from the internal limiting membrane to the nerve fiber layer. Further details of OCTA are available elsewhere.^5,6 

Dataset Processing

The dataset was filtered for missing responses and split into training and testing sets individually for each training instance, which was done 10 distinct times for each model (part of tenfold cross-validation). The total had 193 eyes, but subjects were filtered out for missing responses, less than 33% VF fixation losses, less than 20% false-positive, and less than 20% false-negative responses during the VF examination, leading to 93 POAG and 113 non-glaucomatous control eyes. The VF, structural, and hemodynamic parameters included are listed in Table 1. 

Table 1.

 

The Features Included in Each Feature Set

 

Feature Sets

In our analysis we divided the dataset into eight different subsets termed “feature sets” because they were organized based on biometric features. The subsets include a base feature set containing IOP, HR, SBP, DBP, and VF parameters (Table 1).⁸ All of the following feature sets include the base parameters: a choroidal thickness feature set, an OCT-derived structural biomarker feature set, and an OCTA-derived hemodynamic feature set. Additionally, there were two other feature sets used for comparison: one that combined OCT and OCTA features and a set that included RNFL and GCC only, because these are the gold standard for diagnosis in addition to VF information.⁵ The specific features included in the base, structural, and hemodynamic OCTA-derived feature sets are listed in Table 1, and the feature subsets are listed in Table 2. 

Table 2.

 

The Test AUC From Each Model, Obtained From Tenfold Cross-Validation (SD)

 

Models

MLPs of three sizes were used on each feature set. This allowed accurate comparison of each feature set despite the different number of features, as larger models would be more appropriate for larger feature sets but would be likely to overfit smaller feature sets. Using three models of different sizes offset overfitting and obtained an average AUC. The three MLPs differed in the number of nodes located within the hidden layers. All models had two hidden layers with the following sizes: 5 nodes × 3 nodes, 10 nodes × 3 nodes, and 20 nodes × 10 nodes (Figure), and all the hidden layers used ReLU activation functions. The amount of input nodes is equal to the amount of input features, and the output was always the predicted probability of POAG diagnosis (probability over 0.5 indicated a positive diagnosis) and used a sigmoid activation function. The loss function used for training was Mean Squared Error and the Adam Optimizer was used. 

Figure.

 

Example of neural network models depicted visually: diagram demonstrating the difference between a 5 nodes × 3 nodes MLP and a 10 nodes × 3 nodes MLP. All models were fully connected neural networks with one input layer, two hidden layers, and one output layer. The number of nodes in the input layer varied with each feature set. The hidden layers included 10 nodes then three nodes, 20 nodes then 10 nodes, and five nodes then three nodes. All models had a single output node that included a sigmoid activation to approximate probability of POAG diagnosis. Images generated using NN-SVG.²⁴

AI Analysis

AI analysis was performed using python with Pytorch version 2.1.0. The input data for each model was scaled between 0 and 1 using the Scitkit-Learn standard scaler, which transforms the data using z-score normalization to have a 0 mean and unit variance. Each model was trained and evaluated using tenfold cross validation, which includes splitting the dataset into a training and testing set 10 distinct times to avoid models that exploit statistical trends in the dataset that may not be representative of the real world. Model AUC was assessed by converting the model output into a binary (yes or no) decision. If the model output was equal to or above 0.5, the answer was converted into a 1 (glaucoma positive), and if it was less than 0.5 the diagnosis was 0, or non-glaucomatous. P values were determined by calculating an independent samples t-test using the distribution of accuracies generated by evaluating each model 10 times as a part of tenfold cross-validation. Significance refers to P values less than 0.01. AUC was calculated using the sci-kit learn package. 

Results

After filtering for missing values, 206 eyes were used, 113 without glaucoma and 93 with. Individual measurements were divided into corresponding feature sets (Table 1), and each set of measurements were further divided into a training and test set. Model accuracies were obtained with tenfold cross validation. The following accuracies are reported in Table 2 with standard deviation obtained over the 10 different trials. The average AUC was assessed by averaging across the different model sizes, which measured the AUC obtained by a given feature set regardless of the model size. 

The base feature set reached an average AUC of 0.671. This was the lowest AUC, with only slight improvements achieved with the choroid thickness features which averaged 0.687. The structural set (OCT) reached an AUC of 0.816, the RNFL + GCC only reached 0.764, the hemodynamic (OCTA) set 0.819, the OCTA + OCT 0.809 (Table 2). The Hemodynamic and Structural set had the most features (37); however, the RNFL + GCC set (five features including the IOP, SBP, and DBP) performed comparatively well, indicating that the high diagnostic accuracy was not solely produced through the large number of features. 

Statistical significance calculations were also made comparing the model performances with the structural model. The models that performed insignificantly different include the hemodynamic features, RNFL + GCC only, and the Hemodynamic + Structural. These results indicate that these features worked equally well for training POAG diagnostic AI systems. 

Discussion

The multifactorial nature of POAG necessitates discovery of novel biomarkers and enhanced analysis approaches to improve disease diagnosis and allow tailored therapeutic interventions. Among non-IOP factors, vascular biomarkers have been long implicated; however, their translation into the standard of care remains elusive due to complex interpretation and lack of longitudinal data on predictive utility. 

In this AI analysis MLPs assessed the usefulness of different OCT and OCTA biomarkers for assisting CAD. MLPs of three different sizes were used to account for using inputs that varied in number of features. The smallest feature set contained 10 measurements (IOP + BP + HR + VF) as input whereas the largest set contained 37 measurements as input (Hemodynamic + Structural). The models’ AUC was assessed using a test segment of the dataset that was not used for training. Each model was trained 10 times with 10 different test sets as a part of a tenfold cross-validation. MLPs were used for every model. 

Feature sets in our analysis included IOP + SBP + DBP + HR + VF parameters. The base features combined with the following measurements: structural features including ONH parameters, RNFL and choroidal thickness (OCT-derived), vascular features (OCTA-derived), and RNFL and GCC only. In this analysis, hemodynamic features including OCTA-assessed VD were significantly more accurate for diagnosis than the base model and scored an AUC of 0.819 across the different model sizes and trials. The hemodynamic features had the highest average AUC yet were insignificantly different from the structural OCT measurements, RNFL+GCC, and the Hemodynamic + Structural features. This result implies that OCTA hemodynamic measurements are as useful for computer aided diagnosis as OCT eye-structure measurements. The nature of the statistical relationship exploited by the ML model is unclear, yet this result provides motivation for future studies to explore the statistical relationships between OCTA measurements and glaucoma to uncover how the MLPs were able to reach such a high accuracy. This result supports non-machine learning work by Yarmohammadi et al.,¹² which found that OCTA measurements of VD had a similar diagnostic accuracy for POAG as did measurements of RNFL thickness. 

The RNFL + GCC feature set, which received the base parameters (IOP, SBP, DBP, HR, and VF parameters) with RNFL and GCC had a significantly better accuracy than the base model. This result suggests that RNFL and GCC are useful predictors for POAG diagnosis, which is expected as these structural biomarkers have been well implicated with clinical diagnosis and are commonly relied upon in practice.⁵ Comparing diagnostic accuracy between ML models with different input data suggests that RNFL and GCC results are empirical findings for POAG diagnosis. 

The hemodynamic feature set, the structural feature set, the RNFL + GCC feature set, and the OCTA + OCT feature set all scored accuracies that were significantly better than the base model, and very similar among themselves. This provides evidence that the obtained accuracies did not differ merely because of the number of parameters fed into the model. Even in randomly generated data there exist statistically identifiable trends that machine learning models can exploit. This paradox, known as Simpson’s paradox, was avoided in this case by using tenfold cross-validation, as well as comparing to the RNFL + GCC set, which was much smaller than the OCTA structural set (which includes RNFL and GCC among other parameters, total list in Table 1), which are both smaller than the Hemodynamic + Structural set. Despite vastly different numbers of input parameters, there is no significant increase in performance between these models. Additionally, the hemodynamic model also had a very similar accuracy to these two, indicating that the success of this model was not due only to the use of more input parameters than the base model. 

In this analysis VF data assessed via standard automated perimetry did not lead to models with a high diagnostic AUC of POAG. This was likely due to our patient population being early-stage glaucoma that were diagnosed using structural glaucomatous changes often occurring before significant VF loss. This is an important finding because it shows the efficacy of our method for diagnosing early POAG before significant functional loss. As such, however, these findings may not be applicable to advanced-stage POAG patients who have significant VF defects. These results also serve as a limitation to a MLP approach, which has no way to structure the data and includes no assumptions about the parameters included. It is possible that other ML approaches and larger datasets may be capable of training models based upon early VF changes. 

MLPs are black box models (i.e., they do not provide information about how they use the data for making a diagnosis). Comparing performance across feature sets, however, may be useful for identifying the diagnostic-capability of different features. In this analysis it was a critical assumption that the higher-performing feature sets contained more information about POAG status. This study therefore provides strong evidence that OCTA hemodynamic measurements may enhance AI-driven diagnosis of POAG and motivation for exploring OCTA data for clinical applications. 

In our analysis, the AI models trained with OCTA measurements performed similarly to models that used OCT structural markers for diagnosis,²⁰ and both performed significantly better than the base model. The similarity of success of the structural markers and the hemodynamic models allows for speculation on whether these measurements reflect the same process reported with different units or whether these features show unique aspects of the disease that are distinct and separate. Larger AI studies accessing more data and features inclusive of outcomes over time may reveal further detail of OCTA-assessed VD and its predictability of disease progression. 

Our analysis has several limitations. First, these results are limited by the relatively small size of the dataset and the black-box nature of MLPs. This limits ability to understand if the results are due to a genuine disease biomarker contained in the measurements, or a simple statistic exploit that is not representative of real disease status. Another limitation is that the ML models for this sort of study are only tasked with diagnosing glaucoma from all the information provided, thus the models themselves are unable to provide information regarding the disease process itself. An improvement on this approach could come from physiologically-informed models like that in Guidoboni et al.⁸ The patients examined in this analysis were also early-stage POAG without significant VF loss, and model results may not apply equally across all disease stages. 

Enhancing glaucoma diagnosis and therapeutic avenues before or at disease onset is highly challenging because of individual variance and the multifactorial nature of the disease. In this novel AI ML analysis of VD and POAG status, early-stage glaucomatous disease was predicted by OCTA VD with equal or greater AUC to RNFL and GCC endpoints. Our data suggests that predictability of POAG disease based on baseline structure or function may be enhanced with OCTA assessed of VD. These results further indicate that OCTA VD biomarkers may be highly informative to clinicians at the earliest stages of the disease before significant VF loss. Improvements to POAG diagnosis using AI before or at disease initiation may allow for earlier interventions and more tailored therapeutic approaches to patient care. This analysis suggests longitudinal data combined with MLP approaches that are inclusive of vascular biomarkers alongside IOP and ocular structure may better inform on POAG, providing clinicians a more comprehensive and translatable method to enhance the standard of care. 

Acknowledgments

Supported by NIH grants R01EY030851 (A.H.), R01EY034718 (A.H.), R01EY034718 (J.K.), R01EY034718 (G.G.), R01EY030851 (J.A.), R01EY030851 (B.F.), NYEE Foundation grants (A.H., A.V.V.), The Glaucoma Foundation grant (A.H.), a Challenge Grant Award from Research to Prevent Blindness (A.H., J.C.T..), NSF-DMS 2108711/2327640 (G.G.), NSF DMS-1654019 (J.A.), NSF DMS-2150108 (J.A.), The Glaucoma Foundation (NYC) (L.R.P.), RPB (L.R.P.), and NEI R01 EY 032559 (L.R.P.). 

Disclosure: N. Riina, None; A. Harris, AdOM (C, F, I), Qlaris (C, F, I), Cipla (C), SlitLed (F, I), Oxymap (I), AEYE Health (I); B.A. Siesky, None; L. Ritzer, None; L.R. Pasquale, Twenty-Twenty (C); J.C. Tsai, AI Nexus Healthcare (C), Eyenovia (C), Smartlens (C); J. Keller, None; B. Wirostko, Qlaris Bio (E), MyEyes (C); J. Arciero, None; B. Fry, None; G. Eckert, None; A. Verticchio Vercellin, IRCCS, Fondazione Bietti, Rome (C); G. Antman, None; P.A. Sidoti, Lavelle Fund for the Blind, Inc. (S); G. Guidoboni, Qlaris (C), Foresite Healthcare (C)