Comparison of Retinal Nerve Fiber Layer Thickness Measurement Bias and Imprecision across Three Spectral-Domain Optical Coherence Tomography Devices

Abstract

Purpose.: We compared retinal nerve fiber layer (RNFL) bias and imprecision among three spectral-domain optical coherence tomographs (SD-OCT).

Methods.: A total of 152 eyes of 83 subjects (96 healthy and 56 glaucomatous eyes) underwent peripapillary RNFL imaging using at least 2 of the following 3 SD-OCT devices on the same day: Cirrus HD-OCT (optic nerve head [ONH]) cube 200 × 200 protocol), RTVue-100 (ONH protocol [12 radial lines and 13 concentric circles]), and 3D OCT-1000 (3D Scan 256 × 256 protocol). Calibration equations, bias and imprecision of RNFL measurements were calculated using structural equation models.

Results.: The calibration equations for healthy and glaucoma RNFL thickness measurements among the 3 devices were: Cirrus = 2.136 + 0.831*RTVue; Cirrus = −15.521 + 1.056*3D OCT-1000; RTVue = −21.257 + 1.271*3D OCT-1000. Using Cirrus bias as an arbitrary reference, RTVue bias was 1.20 (95% CI 1.09–1.32, P < 0.05) times larger and 3D OCT-1000 was 0.95 (0.87–1.03, P > 0.05) times smaller. Relative to 3D OCT-1000, the RTVue bias was 1.27 (1.13–1.42, P < 0.05). RTVue imprecision (healthy eyes 7.83, 95% CI 6.43–9.58; glaucoma cases 5.71, 4.19–7.64) was statistically significantly higher than both Cirrus (healthy eyes 3.23, 2.11–4.31; glaucoma cases 3.53, 0.69–5.24) and 3D OCT-1000 (healthy eyes 4.07, 3.11–5.35; glaucoma cases 5.33, 3.77–7.67) in healthy eyes. The imprecision also was significantly higher for RTVue measurements in healthy compared to glaucomatous eyes. None of the other comparisons was statistically significant.

Conclusions.: RTVue-100 showed higher imprecision (or higher measurement variability) than Cirrus HD-OCT and 3D OCT-1000 RNFL measurements. Three-dimensional cube scanning with post-hoc data sampling may be a factor reducing imprecision.

Introduction

Optical coherence tomography (OCT) has become an integral part of ophthalmology practice, especially in the management of glaucoma and retinal disease. A substantial percentage of retinal ganglion cells and their axons are lost before any visual field deficit is apparent in glaucoma. ^1,2 Clinical evaluation of RNFL thickness around the ONH can detect objectively and noninvasively glaucomatous nerve changes and disease progression. ^3–8  

Newer devices using spectral-domain optical coherence tomography (SD-OCT) technology can acquire retinal scans at a speed of up to 200,000 axial scans per second, and a resolution of 3–6 μm through implementation of Fourier transformation and a spectrometer. ⁹ This enhanced data acquisition, compared to an earlier iteration of the OCT technology, also has resulted in an evolution in scan patterns for quantification of the retinal nerve fiber layer (RNFL). Although SD-OCT devices now are used widely across institutions, comparing data taken from these different instruments with various scan patterns is difficult. 

Current scan patterns range from a single circumpapillary circular scan, a combination of circumpapillary radial and circular scans, and three-dimensional (3D) raster scans. These scan patterns differ in the physical scanned region, amount of data collected within the scanned region, and scan duration. It is unknown whether the best measurements come from conventional circumpapillary RNFL scan patterns or other patterns that access more data. Although higher resolution scans potentially can increase the accuracy of RNFL thickness measurements, improve the ability to detect localized focal defects, and track changes over time with high spatial accuracy, the increased time needed to acquire a multitude of data points could be disadvantageous due to artifacts from eye motion and corneal dryness. The purpose of our study was to evaluate the bias and imprecision among different 3D scanning protocols used by three SD-OCT devices to determine the most reliable method for RNFL quantification in healthy and glaucoma subjects. 

Methods

Subjects

Healthy and glaucoma subjects from an academic institute ophthalmic clinic underwent comprehensive ocular examination. Exclusion criteria included history or evidence of retinal pathology, intraocular or keratorefractive surgery (except for uncomplicated cataract surgery), chronic corticosteroid use, or inability to view or image clinically the optic nerve head (ONH) due to media opacity or poorly dilating pupil. All subjects had best-corrected visual acuity of 20/40 or better, and refractive error between +3.0 and −7.0 diopters. All subjects had a reliable visual field (VF), and good quality SD-OCT scans acquired by at least 2 of the following devices: Cirrus HD-OCT (Carl Zeiss Meditec, Dublin, CA), RTVue-100 (Optovue, Fremont, CA), and 3D OCT-1000 (Topcon, Paramus, NJ). All data were acquired at the same visit and both eyes were included if eligible. The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board at the University of Pittsburgh. All subjects signed informed consent before enrollment. 

Glaucoma Subjects

Glaucoma subjects had a glaucomatous-appearing ONH (neuroretinal rim thinning or notch, pallor, or disc hemorrhage), asymmetry in ONH cupping between eyes (cup-to-disc ratio difference >0.2) or RNFL defect, all in the presence of a typical glaucomatous VF (as defined below). 

Healthy Subjects

Healthy subjects had intraocular pressure <21 mm Hg, normal-appearing ONH, symmetric ONH cupping, normal-appearing RNFL, and full VF. 

Visual Field

All subjects had a reliable Humphrey Field Analyzer (Carl Zeiss Meditec) 24-2 Swedish interactive thresholding algorithm (SITA) standard test. A reliable test is one with <33% fixation loss, false positive or negative responses. Qualified normal VFs had a mean deviation (MD) and pattern SD (PSD) within 95% confidence limits of normal reference, and glaucoma hemifield test (GHT) within normal limits. Glaucomatous VF loss was defined as reproducible PSD with a probability of P < 0.05 or GHT outside normal limits, in a pattern consistent with glaucomatous defects. 

SD-OCT Image Acquisition

Subjects were scanned up to two times per eye by at least two instruments in random order during a single visit, with a minimum of 1 minute of rest between scans. Scans were acquired after pupil dilation. Scan quality was ensured by obtaining scans within the manufacturer recommended quality parameter limits, as described below. In addition, scans also were examined subjectively, and those with eye motion larger than one vessel diameter or with regions of lost signal due to blinks were excluded. 

Cirrus HD-OCT

The ONH cube 200 × 200 scanning protocol was used with the Cirrus HD-OCT device (software version 5.1.0.96). This isotropic (equal axial-scan [A-scan] spacing in x and y planes) raster scan contains 200 × 200 × 1024 samplings within a 6 × 6 × 2-mm³ volume scanned at 27,000 A-scans/second, resulting in 40,000 A-scans acquired in 1.48 seconds (Fig. 1). The machine's algorithm automatically identifies the center of the ONH and places a circle with a diameter of 3.46 mm around it. It then extracts 256 A-scan samples from the data cube along this circle and calculates RNFL thickness. Mean global RNFL thicknesses as reported by the machine were recorded. All scans had manufacturer-recommended signal strengths ≥7. ¹⁰  

Figure 1. 

 

Illustration of scan patterns for each SD-OCT device. Cirrus HD-OCT and 3D OCT-1000 perform raster scans. RTVue-100 ONH scan protocol includes 13 concentric circles and 12 radial lines centered on the optic nerve.

RTVue-100

RTVue-100 (software version 6.1.0.4) scans were acquired using the ONH scan protocol. This pattern measures circumpapillary RNFL thickness by recalculating data along a circle of 3.45 mm in diameter around the optic disc, created by a scan pattern made up of 13 concentric circles with diameters from 1.3–4.9 mm, with 0.3 mm intervals and 12 radial lines with diameters of 3.40 mm centered on the ONH (Fig. 1). Sampling rate was 26,000 A-scans/second, resulting in 14,241 A-scans acquired in 0.55 seconds. The software's automatic detection of ONH margins was reviewed and adjusted manually to ensure proper centering over the disc before analysis. Mean global RNFL thicknesses provided by the RTVue system were recorded. All scans had manufacturer-recommended signal strength indices ≥35. ¹¹  

3D OCT-1000

The 3D Scan 256 × 256 protocol, covering a 6 × 6 mm² area, was used with the 3D OCT-1000 device (software version 2.10.2.0, Fig. 1). Using a 3.4 mm diameter circle around the ONH, circumpapillary mean global RNFL thicknesses were calculated automatically by the manufacturer-provided software, which sampled at the rate of 18,000 A-scans/second, resulting in 65,536 scans acquired in 3.6 seconds. All scans had manufacturer-recommended quality (Q)-factors ≥ 45. ¹²  

Statistical Analysis

Measurements recorded by a device reflect a combination of the actual true value (which is unknown) plus device-specific systematic error (bias) and a random error. The SD of the random error is reported as imprecision). Up to 12 measurements (both eyes, both replicates, and three devices) per subject were used to assess simultaneously the relative bias and imprecision. Pairwise scatter plots for mean RNFL are shown in Supplemental Figure 1. A path diagram (Fig. 2) illustrates the structural equation model (SEM) that describes the relationship of the measurements with the unknown true global circumpapillary RNFL thickness values. 

Figure 2. 

 

Path diagram of measurement SEM for comparison of Cirrus HD-OCT, RTVue-100, and 3D OCT-1000. Scale represents relative bias.

The classical measurement error model ^13–16 for measurement X_ij of subject j using device i is:  

 

where α_i and β_i describe how the device systematically distorts the unknown true values μ_j . The μ_j are assumed normal with SD σ. The ε_ij denote random errors that usually are assumed to be normal with mean 0 and SD σ_i . The σ_i describes the device i imprecision. The α_i and β_i describe the bias.  

The method of full information maximum likelihood (FIML) was used to estimate the SEM parameters. This approach allows for missing values so that all 83 subjects could be included. In the SEM, Cirrus was chosen arbitrarily as the reference (path coefficient fixed to 1). The calibration equation relating device pairs is invariant to this choice. The SEM accounts for the expected high correlation between the true mean global RNFL thickness values for each eye and also allows for correlation between replicate pair errors. Replicates for the same device are assumed unbiased and so their path coefficients are fixed to 1. Imprecision SDs for replicates from the same device are assumed to be equal. True values for both eyes are assumed to have the same SD. Although not illustrated in the path diagram, based on initial modeling results, imprecision SDs and correlations were allowed to vary between healthy and glaucoma groups, while systematic components were constrained to be the same for both groups. 

The OpenMx SEM package ¹⁷ for the R language for statistical analysis ¹⁸ was used to compute FIML estimates and likelihood-based CIs for the relative bias and scale-bias-adjusted imprecision SD (imprecision SD divided by the corresponding scale bias) of each device for global circumpapillary RNFL measurements of healthy and glaucoma subjects. The imprecision SDs measure the spread of the random error portion of the measurement error for each instrument, and its units are placed arbitrarily on the same scale as Cirrus for convenience. Finally, the calibration equations relating all pairs of devices were computed using the SEM. P < 0.05 was considered statistically significant. 

Results

We enrolled 152 eyes of 83 subjects (96 eyes of 48 healthy subjects and 56 eyes of 35 glaucoma subjects) into the study. Demographic and clinical data of the subjects are summarized in Table 1. Glaucoma subjects had statistically significantly worse VF and were older than healthy subjects. Due to machine availability, not all subjects were scanned on each device, and the devices were used randomly across all participants. Of the scans 18 (12 on Cirrus, 4 on RTVue, and 2 on 3D OCT-1000) were excluded due to poor quality. The number of scans performed on each device per subject is shown in Table 2. 

Table 1.  

 

Demographics of Healthy and Glaucoma Subjects

	95% CI
	Healthy Eyes (n = 96)	Glaucoma Eyes (n = 56)	Difference	P Value
Age (y)	50.2	67.1	−16.9 (−20.0 to −11.0)	<0.001†
Sex (% female)	68.8	62.9	5.9 (−0.3 to 0.2)	0.75‡
Race (% Caucasian)	89.6	77.1	12.5 (−0.1 to 0.6)	0.21‡
MD (dB)*	−0.01 (−0.95–0.95)	−3.87 (−4.97–−2.77)	−3.87 (−5.32–−2.41)	<0.001§
PSD (dB)*	1.45 (0.89–2.01)	4.10 (3.45–4.75)	2.65 (1.79–3.51)	<0.001§

 dB, decibels.

 * Median.

 † Wilcoxon rank-sum test.

 ‡ Chi-square test.

 § Linear mixed effect model.

Table 2.  

 

Mean RNFL Thickness (μm) of Healthy and Glaucoma Subjects for Each SD-OCT Device

Instrument	Subjects/ Eyes (n)	Mean RNFL Thickness (μm)		P Value*
		Healthy (SD)	Glaucoma (SD)
Cirrus HD-OCT	79/152	92.0 (9.7)	72.5 (13.2)	<0.001
RTVue-100	77/146	109.5 (14.1)	84.7 (15.3)	<0.001
3D OCT-1000	32/62	101.0 (10.9)	81.3 (13.1)	<0.001

 * Student's t-test.

Mean global RNFL thicknesses measured by each instrument are summarized in Table 2. Each device measured a significant decrease in mean RNFL thickness in glaucoma subjects, clearly differentiating glaucoma from healthy patients. RTVue's mean RNFL measurements were 13.6 μm thicker than those of 3D OCT-1000, which were 8.4 μm thicker than those of Cirrus. This “offset” of values represents the systematic error, or bias, of the device. To compare the calibration slopes for healthy and glaucoma groups, we computed the ratio of the glaucoma calibration slope for RTVue as a function of Cirrus to that for healthy eyes, which was 1.006 (95% CI = 0.767–1.341), and similarly for 3D OCT-1000 as a function of Cirrus to that for healthy eyes, which was 0.934 (CI = 0.728–1.171), and RTVue as a function of 3D OCT-1000 to that for healthy eyes, which was 1.078 (CI = 0.782–1.517). All CIs included 1 and, therefore, there was no difference in bias for all three devices between healthy and glaucoma groups, meaning that the systematic measurement differences of values were independent of disease state. Data from healthy and glaucoma groups were pooled for greater power in calculating bias values. 

Using the SEM parameter estimates, calibration equations among the 3 devices were:  

 

 

These calibration curves are illustrated in Figure 3. To test the validity of the formulas, we divided our data randomly in half, and computed calibration equation slopes for RTVue and 3D OCT-1000 relative to Cirrus. The mean ratio for the calibration between the two halves was 1.013 for Cirrus – RTVue and 1.017 for Cirrus – 3D OCT-1000, thus confirming our reported results. 

Figure 3. 

 

Calibration plots of global circumpapillary RNFL thicknesses for Cirrus HD-OCT, RTVue-100, and 3D OCT-1000. Thickness measurements (μm) for pairs of eyes are joined by straight lines to emphasize that the data are clustered. However, for clarity, clustering due to replicate pairs is not similarly shown.

Using Cirrus as a reference for bias estimates, RTVue measurements had a scale bias of 1.204 (95% CI 1.094–1.317), that is 1.204 RTVue units equal 1 Cirrus unit, and this scale bias (relative difference in the size of the measurement units) was statistically significant. The scale bias of 3D OCT-1000 (relative to Cirrus) was 0.947 (95% CI 0.865–1.034). RTVue's scale bias (relative to 3D OCT-1000) was 1.271 (95% CI 1.135–1.422) and was statistically significant. The larger relative scale bias between Cirrus and RTVue, and between 3D OCT-1000 and RTVue, compared to the almost constant bias between Cirrus and 3D OCT-1000, is illustrated in Figure 3. The calibration curve for Cirrus and 3D OCT-1000 is nearly parallel to the diagonal, while the other two calibration curves are nonparallel (and statistically significant) to the diagonal. 

Imprecision of mean global RNFL thicknesses was calculated separately for healthy and glaucoma subjects, since there was a significant difference between the two groups (Table 3). Overall, the RTVue scan pattern had the highest (worst) imprecision, followed by 3D OCT-1000, with Cirrus having the lowest (best) imprecision in both subject groups. The imprecision of RTVue was statistically significantly higher than that of Cirrus in healthy eyes only, and it also was significantly higher than that of 3D OCT-1000 (95% CI 1.35–2.67) in healthy eyes. None of the other comparisons was statistically significant. Comparing imprecision of healthy and glaucoma subjects within each device, RTVue showed significantly lower imprecision in glaucoma subjects compared to its measurements in healthy eyes. In contrast, Cirrus and 3D OCT-1000 showed less imprecision in healthy subjects than in those with glaucoma, although this was not a significant difference. 

Table 3.  

 

Mean RNFL Measurement Imprecision SD (Adjusted for Scale-Bias) and Ratios with Cirrus Values

	Instrument	Scale-Bias-Adjusted Imprecision SD
		Estimate	95% CI	Ratio with Cirrus	95% CI for Ratio*
Healthy	Cirrus HD-OCT	3.23	2.11–4.31	1.00
	RTVue-100	7.83	6.43–9.58	2.42	1.61–4.11
	3D OCT-1000	4.07	3.11–5.35	1.26	0.88–2.14
Glaucoma	Cirrus HD-OCT	3.53	0.69–5.24	1.00
	RTVue-100	5.71	4.19–7.64	1.62	0.89–54.6
	3D OCT-1000	5.33	3.77–7.67	1.51	0.85–54.3

 * CIs for ratios not including 1 are statistically significant.

Discussion

OCT is an important tool in the management of glaucoma. With the advent of faster, newer spectral-domain technologies with different scan patterns, it is difficult to compare data across devices or know which pattern provides the most reliable RNFL thickness measurements. 

In our study, we compared RNFL measurements obtained with SD-OCT devices by using 3D high-resolution raster scan patterns (Cirrus HD-OCT ONH cube 200 × 200 and 3D OCT-1000 3D scan 256 × 256), and a combined radial and circular scan pattern (RTVue-100 ONH) to determine the best scanning strategy. We observed smaller bias and imprecision with raster scans compared to the combined scan pattern. All three devices showed a distinct and significant difference between healthy and glaucoma group RNFL measurements, but there was no difference in bias for all three devices between the two groups, meaning that the systematic measurement differences of values were independent of disease state. 

Measurements from the three devices differed by an average RNFL thickness of up to 17.5 μm. Similar results have been reported previously, ¹⁹ where RNFL thickness measured by RTVue was significantly thicker than that measured by Cirrus (difference of 9.77 μm in a mixed subject pool). Our work also showed increased RNFL thickness measurements by RTVue compared to Cirrus (difference of 17.5 μm in healthy and 12.2 μm in glaucoma subjects). However, the prior study was performed using Bland-Altman plots, and pooled a group of healthy, glaucoma suspects and glaucoma subjects, making it difficult to make a direct comparison between the studies. Statistical evaluation using Bland-Altman analysis has drawbacks when comparing more than two devices because one is forced to do pairwise comparisons, which reduce the power to estimate parameters. Bland-Altman analysis also cannot distinguish whether discrepancies in measurements are due to scale bias or differences in the imprecision without making unsupportable assumptions. We used the SEM analysis, which accounts for bias and imprecision, and avoids the drawbacks of pairwise analysis while allowing to account for the use of measurements from both eyes from some of the participants. Another method used commonly to compare measurements from different devices applies linear regression on measurements from device 1 onto device 2. The disadvantage of this method is that the regression always decreases bias and, thus, underestimates the true relationship. ²⁰ Deming regression is another method that has been used to compare instruments. This method is not affected by the systematic underestimate of the regression coefficient that is seen with standard linear regression. Deming regression has its own drawbacks, most notably the need to have an estimate of the error variance that does not derive from the same data set and is suited best for pairwise comparisons, rather than the three instruments compared in our study. On the other hand, SEM accounts for measurement error coming from multiple instruments, and generates more valid P values. 

Although the manufacturers provide no enabling information, it is plausible that segmentation differences in the definition of the outer border of RNFL are contributing to the systematic bias of each machine. The difference in biases also may be due to the differences in each device's optical interaction with tissue due to different light sources and LCD sensors. Supplemental Figure 2 demonstrates the RNFL delineations on all three devices in a subject with numerically apparent bias. Differences in bias may come from many different variables, including scan patterns, wavelengths, signal strengths, consistency of the centroids of the ONH as defined algorithmically by each instrument, and segmentation algorithms, and it is difficult to generalize specifically from the rough appearance of segmentation outputs. Disease status did not affect the bias, allowing us to compute calibration equations independent of disease severity. These calibration equations can be of utmost value when comparing measurements acquired with different SD-OCT devices over time. 

Reproducibility and imprecision are two closely related but distinct concepts. In most studies, reproducibility is measured by taking repeated measurements of the same subject with the same instrument in similar conditions. This allows the inherent “noise” of the instrument to be gauged, though direct comparison between different devices is not applicable. Several studies reported an excellent reproducibility of measurements obtained by these devices. ^21–23 Imprecision on the other hand, allows the comparison with a “true” value when several devices are used to measure the same structure. This enables a direct comparison of measurements obtained by different devices. High imprecision, as observed in our study, may be caused by reproducibility error, but also may reflect differences in the way an instrument operates when compared to another instrument. Therefore, the significantly higher imprecision of RTVue might be related to the device and not necessarily to the scanning pattern. Yet, the combination of radial and circular scans requires alignment and interpolation, while a raster scan is processed less heavily, which might explain the higher imprecision of RTVue. Scan times, which are another potential source of variability, have a limited effect because the devices with longer scanning time had lower imprecision than those with shorter scanning time. 

Thinning of the RNFL, as seen in our glaucoma subjects, was associated with increased imprecision in Cirrus and 3D OCT-1000 measurements (Table 3). It has been reported previously that glaucoma subjects have poor measurement reproducibility, likely because RNFL measurements of OCT instruments in the lower range are much closer to the noise. ^24,25 Interestingly, we noticed lower imprecision of RNFL measurements by RTVue for glaucoma compared to healthy subjects. Therefore, although RTVue had high imprecision overall, it could be favored when tracking glaucomatous changes in patients with thin RNFLs. On the other hand, to make the distinction between healthy and glaucoma subjects, one of the other instruments with lower imprecisions in the upper range of RNFLs would be preferred. 

In conclusion, RNFL thickness measurements vary among imaging devices. Our study suggests that mean RNFL thickness measurements using the RTVue-100 ONH scan pattern have greater imprecision than the 3D raster scan patterns of 3D OCT-1000 or Cirrus HD-OCT. Three-dimensional cube scanning with post-hoc data sampling may be a factor reducing imprecision. Although there are several differences in the devices' scan protocols and analytical software, which are unable to be controlled for, our analysis of the output of these instruments reflects the current clinically relevant performance of the imaging devices. The effect of scanning pattern warrants further investigation, by using each of the various scan patterns on the same machine, once available.  

Supplementary Materials

doc S1 

Footnotes

 Supported in part by National Institutes of Health contracts R01-EY013178, P30-EY008098 (Bethesda, Maryland), Eye and Ear Foundation (Pittsburgh, Pennsylvania), and an unrestricted grant from Research to Prevent Blindness (New York, New York).

Footnotes

 Disclosure: N.M. Buchser, None; G. Wollstein, None; H. Ishikawa, None; R.A. Bilonick, None; Y. Ling, None; L.S. Folio, None; L. Kagemann, None; R.J. Noecker, None; E. Albeiruti, None; J.S. Schuman, Carl Zeiss Meditec (P)