Effects of Input Data on the Performance of a Neural Network in Distinguishing Normal and Glaucomatous Visual Fields

Abstract

purpose. To compare the performance of neural networks for perimetric glaucoma diagnosis when using different types of data inputs: numerical threshold sensitivities, Statpac Total Deviation and Pattern Deviation, and probability scores based on Total and Pattern Deviation probability maps (Carl Zeiss Meditec, Inc., Dublin, CA).

methods. The results of SITA Standard visual field tests in 213 healthy subjects, 127 patients with glaucoma, 68 patients with concomitant glaucoma and cataract, and 41 patients with cataract only were included. The five different types of input data were entered into five identically designed artificial neural networks. Network thresholds were adjusted for each network. Receiver operating characteristic (ROC) curves were constructed to display the combinations of sensitivity and specificity.

results. Input data in the form of Pattern Deviation probability scores gave the best results, with an area of 0.988 under the ROC curve, and were significantly better (P < 0.001) than threshold sensitivities and numerical Total Deviations and Total Deviation probability scores. The second best result was obtained with numerical Pattern Deviations with an area of 0.980.

conclusions. The choice of type of data input had important effects on the performance of the neural networks in glaucoma diagnosis. Refined input data, based on Pattern Deviations, resulted in higher sensitivity and specificity than did raw threshold values. Neural networks may have high potential in the production of useful clinical tools for the classification of visual field tests.

Perimetry is one the most important examinations for diagnosis and monitoring of glaucoma. Static computerized threshold perimetry in which white stimuli are shown on an evenly illuminated white background has been the most common type of perimetry in clinical glaucoma management for a long time. The way in which perimetric findings are analyzed and presented is important in the interpretation of test results. Reading fields by looking only at maps of numerical threshold sensitivities or gray-scale representation of such values, is difficult even for experts. Programs such as the Humphrey Statpac (Carl Zeiss Meditec, Inc., Dublin, CA) ¹ for computer-assisted interpretation were developed in the mid- to late 1980s. The probability maps included in the Statpac program are often able to highlight early glaucomatous field defects before they become visible in gray-scale representations of raw threshold values ² and can also reduce effects caused by cataract. ³ This probability map concept has enjoyed wide acceptance and has subsequently been applied in most new perimetric devices and perimetric modalities, such as frequency-doubling perimetry ⁴ and short-wavelength automated perimetry. ^{5 6}  

The Glaucoma Hemifield Test (GHT) ⁷ included in Statpac, is a rather simple expert system based on up-and-down hemifield differences between probability scores calculated from Pattern Deviation probability maps. The GHT was one of the first computerized systems that was able to classify field test results reliably as normal or abnormal and improved the ability of ordinary clinicians to assess visual field test results. ⁸  

In the beginning of the 1990s artificial neural networks (ANNs), one of many algorithms in the machine learning classifier concept, were tested as a tool for the interpretation of perimetric results (Goldbaum MH, et al. IOVS 1990;31:ARVO Abstract 2471; Keating D, et al. IOVS 1992;33:ARVO Abstract 1394). ⁹ ANNs were reported to be able to differentiate between glaucoma and normal visual field status at least as well as trained readers. ¹⁰ In other papers, it was also reported that machine learning classifiers discriminate better between normal and glaucomatous fields than do global visual field indices. ^{11 12} Global visual field indices are far from ideal as diagnostic tools, however, because they condense all threshold data into one number, resulting in loss of valuable spatial information, and visual field indices are not particularly sensitive to early localized glaucomatous visual field loss. ^{13 14 15}  

The performance of ANNs has also been compared with that of other types of field interpretation criteria based on localized loss. ¹¹ Disc topography data have also been added to visual field data to improve the diagnostic ability of ANNs. ¹⁶  

We hypothesized that it may be possible to enhance the diagnostic performance of ANNs further by using input data from which the effects of age and media opacities have been eliminated or reduced and in which measured sensitivities have already been compared to the range of age-corrected normal sensitivities and subsequently translated into probabilities. The Statpac program provides two important analyses: (1) Numerical Total Deviations represent the deviation at each tested point of the measured threshold from age-corrected normal values. (2) Numerical Pattern Deviations represent a modification of the Total Deviation results in which a correction has been applied to account for any general elevation or depression of the field caused by media opacities or changes in pupil size. Total and Pattern Deviation probability maps are graphic presentations of the significances of the numerical deviations, relative to the known ranges of normal values at each test point location. 

The purpose of this study was to test our hypothesis by comparing sensitivities and specificities achieved by ANNs for glaucoma diagnosis by using different types of perimetric inputs: numerical threshold values in decibels, and Statpac numerical Total and Pattern Deviations and probabilities. 

Methods

Visual Fields

All visual fields included were obtained with the 30-2 SITA Standard program of the Humphrey Field Analyzer II (model 750; Carl Zeiss Meditec, Inc.). All test point locations in the 30-2 test point pattern were included, except two located in the area of the physiological blind spot. The input data from each field test were:

1.  
   Raw threshold sensitivity values in decibels.
2.  
   Statpac numerical Total Deviations (i.e., deviations in decibels from age-corrected threshold).
3.  
   Numerical Pattern Deviations, which are those same deviations adjusted for the general height of the field and probability scores computed from the Total Deviation and Pattern Deviation probability maps (Fig. 1) . The probability scoring scale was identical with that used in the calculation of the GHT ⁷ —that is, all test points were assigned a score according to significance level of the deviation from the normal value.

Unreliable field test results, defined as a frequency of false-positive answers exceeding 15% or a fixation loss larger than 20% were excluded from the analysis. False-negative rates were not included in the exclusion criteria, because they have been found to correlate highly with the degree of legitimate and reliable glaucomatous field loss measurements. ^{17 18}  

Subjects

Because patients with glaucoma often have concomitant cataract, it is desirable that methods designed to recognize glaucomatous visual field loss not be affected by ocular media changes. Therefore, it was necessary to train the neural networks with fields from healthy subjects, patients with cataract, patients with glaucoma, and patients with both glaucoma and cataract. Patients with glaucoma had primary open-angle glaucoma (POAG), including normal tension, exfoliation, and pigment glaucoma. Other types of glaucoma, such as angle-closure, secondary, and congenital forms were not included. Glaucomatous eyes were defined as those having typical glaucomatous changes in the optic disc: notches, thin or absent neural rims or marked vertical optic cup asymmetry, combined with glaucomatous visual field defects. Glaucomatous field defects were those that were compatible with glaucoma and not explained by other disease. The visual field classification was subjective, including all information available on the single-field printouts. However, we also included seemingly normal fields from eyes with pathologic disc topography, if field defects were found in later visual field tests. Patients with macular or retinal changes and neurologic or endocrinological disorders or other conditions likely to cause field defects were excluded, whereas patients with diabetes mellitus without retinopathy were included. No first field test results of any subjects were considered, to avoid patterns caused by lack of perimetric experience. ^{19 20 21} The study was conducted according to the tenets of the Declaration of Helsinki and was approved by the Ethics Committee of Lund University. 

Healthy Subjects

Two hundred thirteen tests results of 213 subjects were randomly selected from an existing large normal database originally collected to establish normal thresholds and normal limits for the SITA thresholding strategies. ²² The mean age of these subjects was 52 years, ranging from 19 to 84. Most fields in the normative database appeared quite normal, although normality was not a criterion for inclusion; Average Mean Deviation (MD) was −0.02 dB, ranging from −6.11 to +3.07 dB (Fig. 2A) . 

Patients with Media Opacities

These patients had normal disks and normal visual fields and a notation of cataract in their record. We identified 55 such patients. After removing those with unreliable field test results, mostly due to poor fixation, 41 eyes of 41 patients remained in this group. The mean age of these 41 patients was 77 years, ranging from 54 to 96. The MD ranged from −9.82 to −2.46 dB (Fig. 2C) . Forty of these patients had cataract and one had postsurgical opacification of the posterior capsule. These fields were regarded as normal and were included among the 213 normal tests. 

Patients with Glaucoma

The field tests of patients with glaucoma were randomly selected from the directory fields included in the database in one of our Humphrey Field Analyzers. This database consisted of 11,134 tests of 3,629 patients, almost all assessed by the 30-2 SITA Standard program. The directory was sorted in alphabetic order according to the patient’s surname. Starting with the letter A, one field test was randomly selected from every fifth patient; no first field results were selected, to avoid patterns of learning. The selected patients were then matched to our glaucoma register. Only patients with a diagnosis of glaucoma or suspected glaucoma were eligible, and patient records were retrieved. In this way, 643 SITA Standard 30-2 test results were selected to be evaluated for inclusion. At this point the only information available was that the patient had undergone 30-2 SITA Standard visual field testing at least twice, and that the patient had a diagnosis of suspected glaucoma or glaucoma. After retrieving patient records disc photographs obtained before the selected field test were inspected. Fields of all eyes with glaucomatous disc appearance were deemed usable. A comprehensive description of disc topography was required in patient records lacking disc photographs. A description of lens status was also required. The absence of such a description or a notation of a clear lens or pseudophakic eyes was regarded as glaucoma without cataract, whereas data indicating the presence of any type or stage of cataract classified the eyes as having glaucoma plus cataract. After exclusion of eyes according to these criteria, 127 tests of 127 eyes with glaucoma and 68 tests of 68 eyes with concomitant glaucoma and cataract remained. 

The mean age of the 127 patients with glaucoma was 75 years, ranging from 40 to 96. MDs ranged from −31.18 to +0.74 dB (Fig. 2B) . The group with both glaucoma and cataract averaged 77 years of age, ranging from 51 to 97 and had MDs ranging from −29.99 to −0.12 dB (Fig. 2D) . In some eyes, the selected field test results appeared normal, but then the disc appeared suspicious or pathologic, and later field tests, not included in the analysis, showed glaucomatous field loss. 

Neural Network Design

Our networks were fully connected feed-forward multilayer perceptrons built using commercial software (Neural Network Toolbox, ver.4.0 of MatLab; The MathWorks Inc., Natick, MA). This network architecture, consisting of an input layer, two hidden layers, and an output layer, was the same for the different sets of input data. There were 74 units in the input layer, each unit corresponding to one test point in the 30-2 test point pattern. The number of processing elements in the two hidden layers was 25 and 5. The output layer, one neuron with a logistic transfer function, provided the network’s output: glaucoma or normal. 

Network Training

The networks were trained in batch mode by using an optimization of the back propagation algorithm developed by Møller. ²³ This algorithm has been shown to have a fast convergence rate (i.e., relatively few iterations are needed to achieve a small classification error calculated from the network output). An early stopping technique was applied to terminate the training procedure to prevent overfitting of the data. A glaucomatous field classified with 100% certainty was assigned an output of 1 and a 100% normal field an output of 0. Fields falling between were assigned values between 0 and 1. Outputs close to the endpoints 0 or 1 indicated high confidence in the classification, whereas those close to 0.5 indicated uncertainty of the output. Because the task of the network was to identify glaucomatous field loss, patients with cataract only were included in the normal group and patients with concomitant cataract and glaucoma in the glaucoma group. During network training, classification errors were calculated and used to adjust weights in the neural network. The number of necessary iterations, arbitrarily set to a maximum of 300, was also determined by the size of the classification error. Eighty percent of all fields were used in the training procedure. 

Validation

A validation procedure was applied, using half the fields not used in initial training, to prevent overfitting of data. Overfitting of data hampers the network’s generalization ability and effective classification of previously unseen data. 

Evaluation

The performance of the network was evaluated with a 10-fold cross-validation procedure, in which all fields were randomly divided into 10 subgroups each containing 10% of the full data set. ^{12 24} The number of subgroups used in training, early stopping, and test procedures was 8, 1, and 1, respectively. With this procedure, each subgroup was used for training, validation, and evaluation, while ensuring that the network was trained and evaluated, by using different sets of visual fields to avoid confounding. 

Analyses

Network receiver operating characteristic (ROC) curves ²⁵ were produced by adjusting the network threshold. The network threshold, ranging from 0 to 1, was used to define patient classification or diagnosis. For each network threshold, fields with outputs larger than the threshold were classified as glaucomatous, and outputs lower than the network threshold were classified as normal. The areas under the ROC curves, one for each type of input data, were compared by a nonparametric method described by Delong et al., ²⁶ and the Bonferroni correction was applied to adjust for effects of multiple comparisons on the type I error—that is, to reject falsely the null hypothesis stating no difference between ROC curves. 

Results

ROC curves showing combinations of sensitivities and specificities for the different types of data input are shown in Figure 3 . All types of input data formed areas under the ROC curve larger than 0.9, but the different data inputs yielded quite different results. The best results, defined as the largest area under the ROC curve, was achieved with the Pattern Deviation probability scores, followed by the numerical Pattern Deviations. Pattern Deviation probability scores had a significantly larger area under the ROC curve than did threshold sensitivities and numerical Total Deviations and Total Deviation probability scores (P < 0.001). Numerical Pattern Deviations were significantly better (P < 0.001) than both numerical Total Deviations and Total Deviation probability scores, but not better than raw threshold sensitivity values. Threshold sensitivities performed slightly, but nonsignificantly, better than both numerical Total Deviations and Total Deviation probability scores (Fig. 3) . 

In the set of normal fields MDs differed between healthy subjects without (Fig. 2A)and with (Fig. 2C)cataract, whereas the proportion of significantly depressed points in Pattern Deviation probability maps was more similar. For example, the relative number of points depressed at the P < 0.5% level (black box in the probability map) was 0.3% in eyes without cataract and 0.6% in eyes with cataract. The corresponding proportion of such depressed points in Pattern Deviation probability maps in glaucomatous eyes was 26.8% in eyes without cataract and 26.4% in eyes with cataract. 

A best threshold for the network was determined by the best combination of sensitivity and specificity, simply defined as the product of the two. Best network thresholds differed for the various types of input data (Table 1) . 

Discussion

ANNs have been suggested as tools for interpretation of automated visual field test results in patients with glaucoma. ^{10 11} Other types of machine learning classifiers, such as support vector machines or committee machines, have also been reported to interpret visual fields adequately. ¹² In all studies that we have been able to find, however, the inputs have been trained and tested with unprocessed threshold sensitivities. There is no reason to believe that different types of machine learning classifiers would yield different results when different types of input data are compared. We found that using the more refined input data available from a program for computer-assisted interpretation (i.e., Statpac data) could significantly enhance sensitivity and specificity. Pattern Deviation probability scores based on the Pattern Deviation probability maps produced the largest area under the ROC curve, indicating high performance in discrimination between normal and glaucomatous fields. 

The improved results obtained when field data were entered as Pattern Deviations is probably explained by the reduction of the influence of cataract on Pattern Deviations. Both Pattern Deviation numerical displays and probability maps were designed to reduce the effect of media opacities. Pattern Deviation misclassified only 2 normal eyes with cataract, whereas 13 were misclassified when Total Deviation was used. The network was designed to identify the absence or presence of glaucomatous visual field loss. Thus, we included subjects with cataract in the normal group and patients with concomitant cataract and glaucoma in the glaucoma group. We used this approach because cataract frequently occurs in the age groups where glaucoma is most prevalent. 

The normal fields obtained in healthy subjects without cataract were randomly selected from a larger multicenter database used for calculation of Statpac normal values and normal limits for SITA fields. We do not believe that this has biased our results. A large database including data from multiple centers is probably more representative of a normal population than a smaller sample collected at one center only. We did not use the full database; 66% of the records were randomly selected for the purpose of this study. We also included normal fields of patients with media opacities in our set of normal fields. The results, as presented in ROC curves, depended considerably more on the network output than on the Statpac normal limits. Further, our purpose was to compare different input derived from the same normal and pathologic fields and the conclusion pertaining to that comparison would not be expected to cause any bias, as the effects of the selection of the normal data would be equal in all five parameters. 

The five different ANNs correctly classified most fields; but, as expected, normal eyes with substantial cataract were more often classified correctly by the two Pattern Deviation–based ANNs compared with the Total Deviation and unprocessed threshold ANNs (Fig. 4) . In fields with severe damage, Pattern Deviation–based ANNs did not perform as well as ANNs trained with Total Deviation and threshold sensitivities. This was also anticipated, as the Pattern Deviation concept cannot presently be successfully used in end-stage fields. ^{27 28}  

The selection of subjects is crucial when evaluating diagnostic methods. Testing the method in only patients with obvious moderate to severe field defects would give results suggesting better discrimination than would be found in patients with early defects. We randomly selected our glaucoma fields from the directory of tests on the hard disk in one of our perimeters. This resulted in a representative selection of patients with a wide range of visual field defects, including glaucomatous eyes without apparent field loss. With this method, 39% had MDs better than −5 dB and thus could be considered to have mild loss. If only fields with clear-cut reproducible defects were selected, one would expect higher sensitivities for all types of input data. Our selection of fields including a random sample of glaucomatous eyes has advantages, but the selection, in principle, should not be critical when comparing performance of neural networks all using different input data from the same normal and glaucomatous visual fields. 

Our results suggest that the ability of artificial neural networks to classify visual fields can be further improved if refined input data based on Pattern Deviations is used. Such input data resulted in higher sensitivity and specificity than did raw threshold sensitivity values, probably because of the former’s ability to separate field loss caused by glaucoma from that caused by cataract. Further studies including independent visual field data not used for training of network data are needed to evaluate a more general applicability of ANNs for classification of visual field test results. Neural networks and other machine classifiers seem to have a great potential to become a useful clinical tool in the diagnosis of glaucomatous visual field loss, and it may be of value in the study of the performance of a range of types of data inputs with different machine classifiers. 

 Supported by Grant K2002-74X-10426-10A from the Swedish Research Council, by the Järnhardt Foundation, and by funds administered by Malmö University Hospital.

 Submitted for publication February 10, 2005; revised April 14 and May 12, 2005; accepted July 1, 2005.

 Disclosure: B. Bengtsson, None; D. Bizios, None; A. Heijl, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Figure 1.

 

A 30-2 SITA Standard visual field test with thresholds and Statpac Total and Pattern Deviations and corresponding probability maps. This field was correctly classified as glaucomatous by all five types of networks.

Figure 2.

 

Distribution of MDs in (A) healthy subjects (average MD, −0.02 dB), (B) patients with glaucoma (average MD, –9.84 dB), (C) patients with cataract (average MD, −5.27 dB), and (D) patients with concomitant glaucoma and cataract (average MD, −12.13 dB).

Figure 3.

 

ROC curves for each of the five different types of input data. Pattern Deviation (PD) probability scores had the largest area under the curve (0.988), whereas numerical Total Deviations (TD) had the smallest area (0.942).

Table 1.

 

Performance of Neural Network in Classifying Standard Automated Perimetric Visual Fields, using Different Input Data

	Pattern Deviation							Threshold Sensitivity			Total Deviation
		Prob. Scores			dB			dB			Prob. Scores			dB
Network threshold	0.50	0.30 (best)	0.50	0.37 (best)	0.50	0.43 (best)	0.50	0.42 (best)	0.50	0.47 (best)
Sensitivity (%)	89.7	93.9	86.7	90.8	81.5	85.1	79.5	82.1	79.5	80.5
Specificity (%)	97.6	96.5	98.0	94.9	95.3	91.3	94.9	93.3	94.9	94.9
Area under ROC curve	0.988*			0.980, †			0.960			0.943			0.942

*  Significantly better than threshold sensitivity (dB), Total Deviation (dB) and Total Deviation prob. scores (P < 0.001).

†  Significantly better than Total Deviation (dB) and Total Deviation prob. scores (P < 0.001).

Figure 4.

 

Two visual fields correctly classified by the Pattern Deviation numerical and probability score networks, but not with the three networks trained with thresholds and numerical Total Deviation and scores. Left: a field of an eye with healthy disc appearance and cataract; right: a field of an eye with glaucomatous disc appearance and cataract.

The authors thank Ola Engwall, MSc (Lund, Sweden), for performing the necessary programming in the MatLab environment.