Effect of Disease Severity on the Performance of Cirrus Spectral-Domain OCT for Glaucoma Diagnosis

Abstract

Purpose.: To evaluate the effect of disease severity on the diagnostic accuracy of the Cirrus Optical Coherence Tomograph (Cirrus HD-OCT; Carl Zeiss Meditec, Inc., Dublin, CA) for glaucoma detection.

Methods.: One hundred thirty-five glaucomatous eyes of 99 patients and 79 normal eyes of 47 control subjects were recruited from the longitudinal Diagnostic Innovations in Glaucoma Study (DIGS). The severity of the disease was graded based on the visual field index (VFI) from standard automated perimetry. Imaging of the retinal nerve fiber layer (RNFL) was obtained using the optic disc cube protocol available on the Cirrus HD-OCT. Pooled receiver operating characteristic (ROC) curves were initially obtained for each parameter of the Cirrus HD-OCT. The effect of disease severity on diagnostic performance was evaluated by fitting an ROC regression model, with VFI used as a covariate, and calculating the area under the ROC curve (AUCs) for different levels of disease severity.

Results.: The largest pooled AUCs were for average thickness (0.892), inferior quadrant thickness (0.881), and superior quadrant thickness (0.874). Disease severity had a significant influence on the detection of glaucoma. For the average RNFL thickness parameter, AUCs were 0.962, 0.932, 0.886, and 0.822 for VFIs of 70%, 80%, 90%, and 100%, respectively.

Conclusions.: Disease severity had a significant effect on the diagnostic performance of the Cirrus HD-OCT and thus should be considered when interpreting results from this device and when considering the potential applications of this instrument for diagnosing glaucoma in the various clinical settings.

Optical coherence tomography (OCT) provides quantitative in vivo measurements of the thickness of the retinal nerve fiber layer (RNFL) for assessing the structural integrity of the optic nerve. ^1–3 The recent introduction of spectral-domain OCT devices (SDOCT) has greatly enhanced the resolution and scan acquisition times compared with older time-domain versions of this technology and has potentially improved its ability to diagnose and observe the progress of glaucoma. 

In several recent studies, performances of the commercially available SDOCT instruments in detecting eyes with glaucoma have been evaluated, and measures of diagnostic accuracy such as sensitivity, specificity, and receiver operating characteristic (ROC) curves have been reported. ^4–6 However, the accuracy measures reported by these studies have not taken into account the possible effects of important covariates on test results, such as disease severity. Diagnostic tests tend to be more sensitive in advanced stages of the disease, and measures of diagnostic accuracy obtained in studies that include only patients with moderate or severe disease may not be applicable to patients with early disease or those with suspected disease. Further, the comparison of the diagnostic abilities of different tests or parameters may be influenced by the severity of glaucomatous damage. For example, it is possible that a particular parameter is more sensitive at early stages of the disease, whereas another may be more sensitive at moderate or advanced stages. Therefore, it is important to characterize the relationship between the performance of the diagnostic test and the severity of disease and to evaluate how this relationship affects the comparison between different tests. 

The purpose of the present study was to evaluate the effect of disease severity on the performance of one of the SDOCT instruments, the Cirrus HD-OCT (Cirrus HD-OCT; Carl Zeiss Meditec, Inc., Dublin, CA), to diagnose structural damage in glaucoma. 

Methods

This study was an observational case–control design. Subjects included were participating in the longitudinal Diagnostic Innovations in Glaucoma Study (DIGS; Diagnostic Innovations in Glaucoma Study) conducted at the Hamilton Glaucoma Center (University of California, San Diego). Informed consent was obtained from all participants. The University of California San Diego Human Subjects Committee approved all protocols, and the methods described adhered to the tenets of the Declaration of Helsinki. 

Each participant underwent a comprehensive ophthalmic examination, including review of medical history, best corrected visual acuity, slit lamp biomicroscopy, intraocular pressure (IOP) measurement, gonioscopy, dilated funduscopic examination with a 78-D lens, stereoscopic optic disc photography, and automated perimetry with the 24-2 Swedish Interactive Threshold Algorithm (SITA; Carl Zeiss Meditec, Inc.). To be included, subjects had to have best corrected visual acuity of 20/40 or better, spherical refraction within ±5.0 D, cylinder correction within ±3.0 D, and open angles on gonioscopy. Eyes with coexisting retinal disease, uveitis, or nonglaucomatous optic neuropathy were also excluded from the investigation. 

In this study, patients were classified as having glaucoma if they had at least two consecutive and reliable standard automated perimetry (SAP) examinations with either a pattern SD (PSD) outside the 95% normal limits or a glaucoma hemifield test (GHT) result outside the 99% normal limits. Visual fields with false-positive responses, false-negative responses, or fixation losses more than 33% were considered unreliable. In addition, all visual fields were reviewed at VisFACT (Visual Field Assessment CenTer, University of California San Diego). Visual fields were reviewed to identify the presence of artifacts such as lid and rim artifacts, fatigue effects, inattention, or inappropriate fixation. Visual fields were also reviewed for the presence of abnormalities that could indicate diseases other than glaucoma, such as homonymous hemianopia. 

Normal control subjects were recruited from the general population and had IOP < 22 mm Hg with no history of elevated IOP and with at least two reliable normal visual fields, defined as a PSD within 95% confidence limits and a GHT result within normal limits. 

Disease severity was assessed through the visual field index (VFI). Details of the calculation of the VFI have been described elsewhere. ⁷ In brief, the VFI represents the percentage of normal age-corrected visual function, and it is intended for use in calculating rates of progression and staging glaucomatous functional damage. Evaluation of functional loss in glaucoma eyes with the VFI has been demonstrated to be less susceptible than the mean deviation (MD) to the effects of cataract or diffuse media opacities. ⁷ Also, the VFI is thought to more accurately reflect the relative importance of the central and more peripheral visual fields to patient's visual function than do other available global visual function indexes. The VFI can range from 100% (normal visual field) to 0% (perimetrically blind field). 

Cirrus HD-OCT

Spectral-domain OCT was performed with the Cirrus HD-OCT (software ver. 4.0, Carl Zeiss Meditec, Inc.). The device incorporates a super luminescent diode laser with a center wavelength of 840 nm and an acquisition rate of 27,000 A-scans per second that represents up to a 68-fold faster acquisition than time-domain OCT. The available protocol used for RNFL assessment was the optic disc cube. This protocol is based on a tridimensional scan of a 6 × 6-mm² area centered on the optic disc where information from a 1024 (depth) × 200 × 200-point parallelepiped is collected. Then, a 3.46-mm circular scan is placed around the optic disc and the information about parapapillary RNFL thickness is obtained. One advantage of obtaining information about the whole cube is that it is possible to move the circle after the examination is taken, making it possible to manually center it around the optic disc in case of noncentered circles. Because of the shorter acquisition time, measurements with this instrument are theoretically less prone to ocular movement artifacts than is time-domain OCT, leading to more reproducible images. ⁴ To be included, all images were reviewed and had to have a signal strength >7 and the absence of movement artifact. 

Parapapillary RNFL thickness parameters automatically calculated by the Cirrus software and evaluated in this study were average thickness (360° measure), temporal quadrant thickness (316–45°), superior quadrant thickness (46–135°), nasal quadrant thickness (136–225°), inferior quadrant thickness (226–315°), and thickness at each of the 12 clock-hour positions: 3 o'clock, nasal; 6 o'clock, inferior; 9 o'clock; 12 o'clock, superior. 

Statistical Analysis

Descriptive statistics included mean and SD for normally distributed variables and median, first quartile, and third quartile for non-normally distributed variables. ROC curves were constructed to assess the diagnostic ability of the Cirrus HD-OCT in discriminating glaucomatous eyes from normal eyes for each parameter. The ROC curve provides the tradeoff between the sensitivity and 1 − specificity. The area under the ROC curve (AUC) was used to summarize the diagnostic accuracy of each parameter. An AUC of 1.0 represents perfect discrimination, whereas an area of 0.5 represents chance discrimination. 

Initially, we report pooled AUCs for each parameter adjusted only for age differences between glaucomatous and healthy eyes. Then, an ROC regression modeling technique was used to study the effect of disease severity on the diagnostic accuracy of the Cirrus HD-OCT parameters. This approach was applied by Medeiros et al. ⁸ for evaluation of the influence of covariates on the performance of diagnostic tests in glaucoma. This methodology allows the evaluation of the influence of covariates on the diagnostic performance of the test, so that ROC curves for specific values of the covariate of interest can be obtained. Also, it allows adjustment for the possible confounding effects of other covariates. Details of the modeling procedure have been described previously. ^8,9  

In the present study, an ROC regression model was fitted, to investigate the influence of disease severity (as measured by the VFI) on the diagnostic ability of the Cirrus HD-OCT. Briefly, the model can be written as:  

 

where ROC _X,XD (q) is the probability that a subject with disease, under the effect of a disease-specific covariate, X _D, and a common covariate, X, will have results that are greater than or equal to the qth of the distribution of test results from subjects without disease. That is, when the specificity of the test is 1 − q, ROC _X,XD (q) represents sensitivity. The intercept and the slope of the ROC curve are represented by the coefficients α₁ and α₂, respectively. Φ represents the normal cumulative function. If the coefficients (β) for disease-specific (X _D) or common (X) variables are greater than 0, then the variable positively influences the discrimination between subjects with disease and between normal and diseased, respectively. A negative coefficient indicates an inverse relationship. Interaction terms between the variables and Φ⁻¹(q) can be included to allow the effects of the covariates to differ by various amounts, depending on the false-positive rate q (or specificity 1 − q), that is, to influence the shape of the curve. 

In our study, the ROC regression model was fitted with VFI used as the disease-specific covariate. Because of its potential influence on RNFL thickness, ¹⁰ our ROC regression model was adjusted for age. After calculating all coefficients from the ROC regression method in a multivariable model, an estimation of the AUCs for arbitrary values of VFI was obtained using the formula:  

 

Parameters were estimated by using probit regression. To obtain confidence intervals for regression parameters, a bootstrap resampling procedure was used (n = 1000 resamples). ¹¹ As measurements from both eyes of the same subject are likely to correlate, the use of standard statistical methods for parameter estimation can lead to underestimation of standard errors and to confidence intervals that are too narrow. ¹² Therefore, to account for the fact that both eyes of some subjects were used for analyses, the cluster of data for the study subject was considered as the unit of resampling when calculating standard errors. This procedure has been used to adjust for the presence of multiple correlated measurements from the same unit. ^11,13  

All statistical analyses were performed with commercially available software (Stata version 10; StataCorp, College Station, TX, and SPSS, ver 16.0; SPSS Inc, Chicago, IL). The α level (type I error) was set at 0.05. Our study had an 85% power to detect a difference of at least 0.1 between an area under the ROC curve and 0.5, which represents chance discrimination. 

Results

The study included 135 eyes of 99 patients with glaucoma and 79 eyes of 47 normal control subjects. Demographic and clinical characteristics are shown in Table 1. The average VFI score of the 135 glaucomatous eyes was 85.6% (median, 92%; first quartile, 84%; third quartile, 96%). Figure 1 shows the distribution of VFI scores in the glaucomatous eyes included in the study. There was a large variation in VFI in the glaucomatous eyes (ranging from 22% to 100%). The glaucomatous eyes had an average MD of −5.63 dB (median, −3.66 dB; first quartile, −6.76 dB; third quartile, −2.06 dB). 

Table 1.

 

Demographic and Ocular Characteristics of the Glaucoma and Healthy Control Study Groups

	Glaucomatous Eyes (n = 135)	Normal Eyes (n = 79)	P
Age,* y	66 (11.65)	60 (11.7)	0.016
Sex, % male	46	34	0.035
Visual field MD,† dB	−5.63 (−6.76, −3.66, −2.06)	0.06 (−0.44, 0.135, 0.82)	<0.001
Visual field PSD,† dB	5.67 (2.5, 4.03, 9.02)	1.49 (1.29, 1.47, 1.66)	<0.001
VFI,† %	85.59 (84, 92, 96)	99.36 (99, 100, 100)	<0.001
Spherical equivalent, D*	−0.50 (1.74)	−0.14 (1.89)	0.262

*  Normally distributed variables; represented by mean (standard deviation).

†  Non-normally distributed variables; represented by mean (first quartile, median, third quartile).

Figure 1.

 

Frequency distribution of VFIs in the 135 glaucomatous eyes.

Table 2 shows the areas under the pooled ROC curves for each parameter of the Cirrus HD-OCT. These areas represent age-adjusted results from the population sample without the influence of disease severity. The RNFL thickness parameters with the largest AUCs for discriminating glaucomatous from healthy eyes were the average thickness (0.892; 95% confidence interval [CI], 0.843–0.942), inferior quadrant thickness (0.881; 95% CI, 0.826–0.935), and superior quadrant thickness (0.874; 95% CI, 0.815–0.933). 

Table 2.

 

Age-Adjusted Areas under the “Pooled” ROC Curves for Cirrus HD-OCT RNFL Thickness Parameters

Parameter	AUC	95% CI
Average thickness	0.892	0.843–0.942
Inferior quadrant	0.881	0.829–0.932
Superior quadrant	0.874	0.817–0.931
7 o'clock	0.863	0.805–0.920
6 o'clock	0.844	0.782–0.904
11 o'clock	0.821	0.752–0.889
8 o'clock	0.802	0.729–0.875
12 o'clock	0.787	0.710–0.863
Temporal quadrant	0.757	0.671–0.843
10 o'clock	0.751	0.667–0.835
1 o'clock	0.751	0.674–0.828
5 o'clock	0.718	0.634–0.799
2 o'clock	0.651	0.553–0.747
Nasal quadrant	0.648	0.551–0.745
4 o'clock	0.610	0.514–0.706
9 o'clock	0.608	0.513–0.703
3 o'clock	0.597	0.501–0.692

Table 3 shows the results of the ROC regression model investigating the effect of disease severity on the accuracy of the Cirrus parameter average thickness. The severity of disease, as measured by the VFI, had a significant influence on the diagnostic performance, as indicated by the statistically significant value attributed to the coefficient representing severity (β₁ = −0.038; 95% CI, −0.072 to −0.016). As expected, this parameter performed better in patients with more severe disease. There was a tendency for disease severity to exert a relatively greater effect on lower false-positive rates (i.e., higher specificities), as indicated by the negative coefficient for the term: Severity × Φ⁻¹(q). However, this correlation was not statistically significant (β₂ = −0.006; 95% CI, −0.043 to 0.08). Figure 2 shows ROC curves for arbitrary values of VFI. The areas under the ROC curves for arbitrary VFIs of 100%, 90%, 80%, and 70% were 0.822, 0.886, 0.932, and 0.962, respectively (Table 4). Figure 3 shows sensitivities at fixed specificities for detection of glaucoma according to levels of disease severity for the average thickness parameter. As expected, sensitivities were significantly higher for worse disease severities. A similar effect was observed when using MD as an indicator of disease severity in our ROC regression model. We obtained AUCs of 0.848, 0.917, and 0.959 for MD values of 0, −6, and −12, respectively. Using PSD as an indicator of severity, we found AUCs of 0.801, 0.932, and 0.983 for values of 0, 6, and 12 dB, respectively. 

Table 3.

 

Results of the ROC Regression Model to Investigate the Effect of Disease Severity on the Diagnostic Performance of the Average RNFL Thickness Parameter

Parameter	Coefficient	Estimate	95% CI
Intercept	α1	4.597	(2.889 to 8.194)
Φ−1 (q)	α2	1.292	(0.018 to 4.801)
Severity	β1	−0.038	(−0.072 to −0.016)
Severity × Φ−1(q)	β2	−0.006	(−0.043 to 0.08)

Figure 2.

 

ROC curves for the RNFL average thickness parameter for arbitrary VFIs, according to the regression model.

Table 4.

 

AUC for Selected RNFL Thickness Parameters and Arbitrary Values of Disease Severity, According to the Regression Models

VFI (%)	Average	Inferior	Superior
100	0.822 (0.761–0.882)	0.851 (0.783–0.920)	0.812 (0.738–0.877)
90	0.886 (0.823–0.949)	0.871 (0.815–0.934)	0.874 (0.812–0.936)
80	0.932 (0.883–0.982)	0.920 (0.874–0.972)	0.921 (0.870–0.974)
70	0.962 (0.915–1.00)	0.954 (0.902–1.00)	0.952 (0.897–1.00)

 Data are AUCs for RNFL parameters (95% CI).

Figure 3.

 

Sensitivities at fixed specificities for detection of glaucoma according to levels of disease severity for the average thickness parameter.

Similar results were found when the effect of disease severity was evaluated on the other Cirrus parameters (Table 4). For early stages of disease, the parameter inferior quadrant thickness performed better than average thickness, a result that was not suggested by simple inspection of the pooled ROC curves. 

Discussion

In the present study, we demonstrated that disease severity significantly affects the performance of the Cirrus HD-OCT for detection of glaucoma. By incorporating VFI as a measure of disease severity in ROC regression models, we demonstrated that, in RNFL assessment, this instrument performed significantly better in detecting damage in subjects with more severe visual field loss. To the best of our knowledge, this study is the first in which the effect of disease severity on the diagnostic performance of the Cirrus HD-OCT was evaluated, and our results may have significant implications for the use of SDOCT in glaucoma diagnosis. 

Diagnostic tests tend to be more sensitive in advanced stages of the disease and measures of diagnostic accuracy obtained from studies that include only patients with moderate or severe disease may not be applicable to patients with early-stage or suspected disease. ^14,15 We found a pronounced effect of disease severity on the performance of the different Cirrus RNFL thickness parameters. In patients with minimal visual field losses or VFI of 100%, the AUC for the parameter average thickness was 0.822, with sensitivity of 47.9% and specificity at 95%. The AUC increased to 0.962 in patients with VFI of 70%, with sensitivity of 84% and specificity at 95%. Understanding the relationship between disease severity and test performance is important in evaluating the potential applications of a test in different clinical scenarios. For example, from the results of our study, it can be expected that the performance of the Cirrus HD-OCT in screening for patients with severe disease will be different from that for the identification of subjects with early glaucomatous damage in clinical practice. 

It is interesting to note that the performance of the Cirrus OCT in detecting damage in glaucomatous eyes with VFI of 100% is similar to that of other imaging instruments in the diagnosis of preperimetric glaucoma. Medeiros et al. ¹⁶ found an area under the ROC curve of 0.89 for diagnosis of preperimetric glaucoma with scanning laser polarimetry using enhanced corneal compensation. It should be noted, however, that the diagnosis of preperimetric glaucoma in other studies was performed on the basis of documented evidence of optic disc progression in patients without any evidence of visual field loss. In the present study, all included subjects had evidence of repeatable visual field abnormalities, although some had minimal defects and VFI still of 100%. Further studies should be conducted for longitudinal evaluation of the performance of Cirrus HD for detection of preperimetric glaucomatous damage. 

The analysis of pooled ROC curves for discriminating glaucomatous from normal patients revealed that the parameters average thickness, inferior quadrant thickness, and superior quadrant thickness had the largest areas under the ROC curves. Leung et al. ⁴ described these same RNFL parameters as having the greatest diagnostic accuracy; however, larger AUCs were obtained in their work. For the average thickness parameter, Leung et al. reported an ROC curve area of 0.962 versus 0.892 in our study. The difference in diagnostic performance may be explained by differences in the glaucoma severity in the patients included in the two studies. The average MD of the visual field in our glaucoma population was −5.6 dB compared with −8.66 dB in the study by Leung et al. In fact, using MD instead of VFI as a covariate in the ROC regression model, we obtained an AUC of 0.932 for average thickness for an MD level of −8.7 dB, very similar to the result found by Leung et al. These findings highlight the importance of investigating and reporting the effect of disease severity on studies of diagnostic performance. 

The comparison of the diagnostic abilities of different parameters may also be influenced by the severity of the disease. It is possible that a particular test may be more sensitive at early stages of the disease, whereas another test may be more sensitive for moderate or advanced stages. In our study, we found that, in early stages of disease, the AUCs for the inferior average parameter were larger than those for average thickness and superior average. This result is in agreement with the expected pattern of initial glaucomatous optic nerve damage. In patients with severe disease, no differences were observed in the performance of the different RNFL parameters. In these cases, the degree of damage is such that differentiating them from healthy eyes is an easier task. 

Other investigators have also evaluated the influence of covariates on the performance of diagnostic tests in glaucoma. Based on a logistic regression model proposed by Leisenring et al., ¹⁷ Stroux et al. ¹⁸ evaluated the influence of disease severity on the sensitivities of several different visual function and electrophysiological tests. The logistic model developed by Leisenring et al., however, was originally proposed for evaluation of tests with categorical results. Therefore, the evaluation of tests with continuous results using this approach requires that the test results be dichotomized according to arbitrary cutoffs of specificity or sensitivity. The method used in the present study was initially used by Medeiros et al. ⁸ to evaluate the performance of frequency-doubling technology in glaucoma. It is advantageous, as the effects of covariates can be assessed on the whole ROC curve and, therefore, do not require dichotomization of test results. We have also used the current methodology to assess the effect of disease severity on the diagnostic performance of scanning laser polarimetry in glaucoma. ¹⁹  

Other covariates may also influence the diagnostic accuracy of an instrument. Our glaucomatous patients were, on average, 6 years older than normal control subjects. A study by Parikh et al., ¹⁰ showed that the RNFL thickness becomes thinner with advancing age. To account for the potential influence of age on RNFL thickness, our pooled ROC and ROC regression models were adjusted for age. In the present study, glaucomatous eyes had a mean signal strength of 7.9, significantly lower than the mean signal strength of 8.9 obtained from normal eyes (P < 0.001). Previous studies have shown that the magnitude of the signal strength is associated with RNFL thickness ^20–22 ; however, these studies were performed with time-domain OCT and included eyes with relatively lower signal strength. In our study, we included only eyes with signal strength greater than 7. By selecting only good-quality images, we expect to minimize the effect of signal strength on RNFL thickness. In fact, in a multivariate model including age, signal strength, and diagnostic group as independent variables and RNFL thickness as a dependent variable, signal strength had no influence on average RNFL thickness. 

We used the VFI as a descriptor of disease severity in our study. There is no consensus with regard to the best staging classification system for glaucoma patients; however, using the VFI as a covariate to represent disease severity has some advantages when compared with other proposed indexes such as MD or PSD. MD can be highly influenced by media opacities, and PSD may miss advanced visual field damage in glaucoma. ²³ Categorical staging systems, such as the glaucoma severity staging (GSS), ²⁴ can be readily understood in clinical practice, but the categorization of what is in fact a continuous measure causes a loss of statistical power. 

Our study had some limitations. Although we included patients with a broad spectrum of disease severity, most of our patients had early visual field defects that may have resulted in a lesser influence of disease severity on our ROC regression model than if we had used a larger proportion of patients with severe disease. Further, all our glaucoma patients had evidence of visual field loss, and our control population constituted healthy individuals with no suspicious findings for disease, which may have increased the performance of the test. In fact, longitudinal studies should be conducted to evaluate the performance of Cirrus HD-OCT in diagnosing damage in subjects with suspected glaucoma. 

In conclusion, we demonstrated that disease severity has a significant effect on the diagnostic performance of the Cirrus HD-OCT. These findings should be considered when interpreting results from this device and also when considering the potential applications of this instrument for diagnosing glaucoma in different scenarios. 

Footnotes

 Supported in part by CAPES-Brazil Grant BEX1327/09-7 (MTL), National Eye Institute (NEI) Grant EY08208 (PAS), and NEI Grant EY11008 (LMZ). Participant retention incentive grants in the form of glaucoma medication at no cost were provided by Alcon Laboratories Inc., Allergan, Pfizer Inc., and Santen, Inc.

Footnotes

 Disclosure: M.T. Leite, None; L.M. Zangwill, Carl Zeiss (F), Heidelberg Engineering (F); R.N. Weinreb, Carl Zeiss (F, C, R), Optovue (F, C), Heidelberg Engineering (F, C, R), Topcon (F, C); H.L. Rao, None; L.M. Alencar, None; P.A. Sample, Carl Zeiss (F), Heidelberg Engineering (F); F.A. Medeiros, Carl Zeiss (F, R), Heidelberg Engineering (F, R)