Repeatability of Stratus Optical Coherence Tomography Measures in Neovascular Age-Related Macular Degeneration

Abstract

purpose. To determine the repeatability of Stratus optical coherence tomography (OCT) measures of retinal thickness and volume in patients with neovascular age-related macular degeneration (nAMD)

method. Fifty-one eyes of 51 consecutive patients with nAMD underwent an OCT imaging session in which two fast macular thickness map (FMTM) protocol scans sets were acquired by a single experienced operator certified for clinical trials work. Coefficients of repeatability for each of nine Early Treatment of Diabetic Retinopathy Study (ETDRS)-like regions, foveolar center-point retinal thickness (CPT) and total macular volume (TMV), were calculated. Scans were analyzed retrospectively for errors in retinal boundary placement by two observers, with revised coefficients of repeatability calculated after excluding any scan sets with significant segmentation error.

results. The coefficient of repeatability for the central 1-mm macular subfield was 67 μm (23%) and was less than 75 μm for all macular subfields. There was much larger variability in the center-point thickness measure, with a coefficient of repeatability of 88 μm (32%) for the automated center-point thickness (ACPT). After excluding nine scan set pairs with significant segmentation error, the coefficient of repeatability for the central 1-mm macular subfield was reduced to 50 μm (19%).

conclusions. OCT-derived retinal thickness measurements are subject to considerable measurement variability in patients with nAMD. Changes in central macular thickness of more than 50 μm may better reflect true clinical change in scan sets without significant segmentation error and may be used to guide the retreatment of patients with nAMD in clinical trials and clinical practice.

Optical coherence tomography (OCT) is a noninvasive technique providing optical cross-sections of the retina. ¹ The high resolution of the images and the ability of OCT to quantify changes in retinal thickness have led to the technique assuming a central role in the evaluation of the patient with neovascular age-related macular degeneration (nAMD), both in assessing the response to treatment and in the need for retreatment. ²  

With the arrival of intravitreal agents that block the actions of vascular endothelial growth factor-A (VEGF), there has been a move from a qualitative evaluation of OCT findings in nAMD to the quantitative assessment of macular thickness and assessment of the change in retinal thickness as a response to treatment. The most favored protocol to assess quantitative change in macular thickness is the fast macular thickness map protocol (FMTM), which allows rapid image acquisition (less prone to movement error) with low image resolution (128 A-scans per optical section). 

To date, both reading center-determined eligibility for clinical trials and investigator-determined retreatment decisions are based on changes in clinical, fundus fluorescein angiography (FFA), and OCT parameters associated with choroidal neovascularization (CNV) activity. The repeatability of retinal thickness measurements in patients with nAMD is unknown. Providing an estimate of the repeatability of OCT-derived retinal thickness measurements in these patients will help to determine the degree of change in retinal thickness measurement that may better represent true clinical change rather than measurement variability. This may then be used to aid retreatment decisions in future clinical trials and clinical practice. 

The difficulty in estimating the repeatability of automated retinal thickness measurements in nAMD arises from the significant artifacts and errors that affect scans in this group of patients. ³ These errors include misplacement of inner and outer retinal boundary lines by the StratusOCT (Carl Zeiss Meditec, Inc., Dublin, CA) computer algorithm (segmentation error), which will contribute to the variability of automated retinal thickness measurements. In this study, we noted any FMTM scan sets with all six line scans affected by retinal boundary errors, enabling us to report revised coefficients of repeatability for retinal thickness measurements after excluding these pairs of scan sets with significant segmentation error. 

The patients used to determine estimates of repeatability of retinal thickness measurements have included healthy persons ^{4 5 6} and diabetic patients, ^{4 6} but, to our knowledge, this is the first study to provide an estimate of repeatability of StratusOCT measurements in patients with nAMD. 

Subjects and Methods

Equipment

Optical coherence tomography is an imaging technique that uses the analysis of optical interference patterns of low coherence and wide bandwidth light to generate optical cross-sections of the retina. ¹  

All OCT imaging was performed using the commercially available StratusOCT machine (Carl Zeiss Meditec Inc.) with software version 4.0. This is the third generation of OCT machine and provides an axial resolution of less than 10 μm. The OCT machine is serviced regularly, in line with manufacturer recommendations, by authorized technicians evaluated by authorized technicians and personnel from Carl Zeiss Meditec Inc. to ensure that the machine is calibrated and operating correctly. 

Subjects

OCTs of 51 eyes of 51 consecutive patients with nAMD undergoing evaluation and treatment with anti-VEGF agents were analyzed by searching the StratusOCT database at the Clinical Trials Unit at Moorfields Eye Hospital. Approval for the collection and analysis of OCT images was obtained from the Research Governance Committee of Moorfields Eye Hospital. 

All patients in this study had subfoveal CNV caused by AMD in the study eye and either had undergone or were about to undergo treatment. For each patient, only images from the eye undergoing treatment were used in the analysis. 

For this study, all scans were obtained between May 8, 2007, and June 15, 2007, on a single StratusOCT machine. All imaging was performed after pupil dilation with one drop of 2.5% phenylephrine hydrochloride and 1% tropicamide. Patients consented to OCT imaging as part of their clinical care or as part of clinical trial involvement. The research followed the tenets of the Declaration of Helsinki. The scans are from consecutive patients at different stages of treatment and represent a wide range of disease activity and retinal thicknesses, with some patients with well-treated, quiescent lesions and relatively normal retinal thicknesses and other patients with active CNV and gross retinal thickening. 

Scanning

All scanning was performed by a single technician certified by image reading centers for OCT scanning in pharmaceutical company-sponsored AMD clinical trials. In line with other studies, the FMTM protocol was used to assess retinal thickness. The FMTM protocol uses six high-speed, 6-mm radial lines (oriented 30° apart) to delineate macular anatomy and pathology. The protocol enables all six line scans to be acquired in a continuous, automated sequence within 1.92 seconds, with each of the six lines composed of 128 equally spaced transverse sampled locations (total of 128 × 6 lines or 768 sampled points). At each of these locations, the signal is sampled axially at 1024 equal intervals over a depth of 2 mm. In this study, the term “scan set” refers to all six line scans acquired automatically as part of the FMTM protocol, and a single “scan” refers to an individual line scan. Because of its short acquisition time, the FMTM protocol is believed to be less prone to errors caused by unstable fixation, an important consideration in the assessment of the patient with nAMD. This rapid acquisition of scans is, however, at the expense of scan resolution in the transverse plane. 

For each patient, two consecutive FMTM scan sets were used for analysis. Consecutive scan sets were acquired in a single imaging session, in line with departmental and clinical trials protocols. The patient was aligned correctly with the OCT machine and was asked to look at an internal fixation light. If no light was seen, the patient was asked to look straight ahead by use of an external fixation light to ensure the scans were taken through the fovea. The technician was experienced in identifying common artifacts in OCT images, and scan sets were reacquired as needed to optimize scan quality. Low-quality scans with a signal strength less than 7, scans with artifacts, and scans with a low analysis confidence message from the onboard StratusOCT software were discarded, and the technician was instructed to save only high-quality, well-centered scans with signal strength greater than 6, though such standards were not always achieved. Once image quality had been optimized, the technician acquired two FMTM scan sets in the same session, with the patient sitting back from the machine between scan sets but the second scan set acquired without delay to minimize patient fatigue and additional fixation losses in the second scan set. Each scan was analyzed using the onboard StratusOCT software (version 4.0) with segmentation of the retinal layers and quantitative measurement of retinal thickness and volume. The macular thickness map analysis uses data from all six linear scans to interpolate the retinal thickness measurements across circular Early Treatment of Diabetic Retinopathy (ETDRS) sectors with nine sectoral thickness values for circles with diameters of 1, 3, and 6 mm (Fig. 1) . The automated measurements (in mm²) of retinal thickness in each of the nine ETDRS subfields were recorded from the macular thickness map analysis, as were the automated center-point thickness (ACPT) and total macular volume (TMV) in mm³. In addition, a manual measure of the center-point retinal thickness (MCPT) was taken in line with the 64th A-scan for each of the six individual line scans in each of the two scan sets by an observer certified for work in pharmaceutical company-sponsored AMD clinical trials (PJP). 

Because retinal boundary placement error may significantly affect the variability of the automated measures of retinal thickness, a note was made of the number of scan sets with significant segmentation error (in which at least one scan set contained all six line scans affected by retinal boundary errors as recorded by two independent observers experienced in AMD clinical trials work (PJP and FKC)). A retinal boundary placement error was defined by the authors as a visible difference between computer algorithm-determined and observer-determined inner or outer retinal boundaries (at least 16 μm using StratusOCT caliper determined vertical distance). This would then allow a recalculation of coefficients of repeatability excluding any pairs of scan sets with significant segmentation error. 

Statistical Analysis

We calculated the coefficient of repeatability (1.96 × SD of differences between pairs of measurements in the same subjects by the same technician at the same visit) in line with the methods outlined by Bland and Altman ⁷ for each of the automated retinal thickness measurements in the nine ETDRS subfields, the ACPT, and the TMV. In addition, we calculated the coefficient of repeatability for the MCPT. To allow comparison with other studies, we also expressed the coefficients of repeatability as a percentage of the mean measurement for each category of measurement (this equates to 1.96 × the coefficient of variation). Mean values were also calculated for the differences in retinal thickness and volume measures. The Wilcoxon signed rank test (5% significance level) was performed to determine any statistically significant difference between the measurements from the two scan sets. Bland-Altman plots were also constructed to investigate any trend for higher variability at greater macular thicknesses. 

We also recalculated the coefficient of repeatability for all measures of retinal thickness and TMV after excluding scan sets with significant segmentation error. 

Results

Excluded Scans

Only 50 pairs of scan sets were analyzed; OCT scans of one patient were vertically truncated because of excessive pigment epithelial detachment height, preventing a meaningful measurement of retinal thickness. We did not exclude scan sets with automated segmentation error because this is known to occur in a significant proportion of patients with neovascular AMD undergoing OCT imaging ³ and forms part of the variability of automated retinal thickness measurements. We did, however, perform a subanalysis excluding scan set pairs in which at least one scan set contained all six line scans affected by segmentation error (significant segmentation error). 

Subject Characteristics

The mean age (± SD) of the 50 patients with nAMD was 80 ± 7 years (range, 64–91 years). There were 28 women and 22 men. Forty-nine (98%) of the patients were Caucasian. 

Automated and Manual Measures of Retinal Thickness and Automated Measure of Total Macular Volume

Medians and interquartile ranges for retinal thickness and volume measurements are shown in Table 1 . The Wilcoxon signed rank test (5% significance level) showed no statistical difference between the intrasession measurements. Figures 2 3 and 4show Bland-Altman plots for the mean central 1 mm macular (A1) subfield, the ACPT, and the MCPT, respectively. Other plots for macular subfields A2 to A9 and TMV looked similar. None of the plots suggested a relationship between repeatability and mean retinal thickness or volume. Table 2presents repeatability coefficients expressed in microns and as a percentage of the mean thickness or volume. The coefficient of repeatability was less than 75 μm (26%) for all macular subfields (A1-A9) with a value of 67 μm (23%) for the central 1-mm (A1) subfield, the commonly reported macular subfield in clinical trials. The least repeatable measure was the ACPT measurement, with a coefficient of repeatability of 88 μm (32%). 

Scan Sets with Significant Segmentation Error

Both observers (PJP, FKC) agreed there were nine pairs of scan sets in which at least one scan set was affected by retinal boundary errors in 6 out of 6 line scans (significant segmentation error). 

Revised Intrasession Coefficients of Repeatability for Scan Sets without Significant Segmentation Error

Excluding the nine pairs of FMTM scan sets with significant segmentation error, 41 pairs of scans were left for subanalysis. The revised coefficients of repeatability were 76 μm (28%) or less for all macular subfields (A1-A9) with a value of 50 μm (19%) for the central 1-mm (A1) subfield. The coefficients of repeatability for the center-point thickness measures were 66 μm (26%) for ACPT and 68 μm (27%) for MCPT (Table 3) . 

Discussion

This study provides an estimate of the repeatability of StratusOCT-derived retinal thickness and volume measurements in patients with nAMD. In addition, revised coefficients of repeatability are presented after nine scan set pairs with significant segmentation error were excluded from the analysis. Strengths of the study included the range of retinal thicknesses included in the analysis and the fact that the scans were from consecutive patients with nAMD undergoing treatment. It is difficult to provide a measure of intersession reproducibility of retinal thickness measurements for patients with nAMD because the disease may progress rapidly, causing real change in retinal thickness. In addition, because these patients are treated promptly, there is little opportunity to repeat OCT imaging in a different session before treatment. 

We report a repeatability coefficient of 67 μm (23%) for the central 1-mm (A1) macular subfield. After identifying and excluding nine pairs of scan sets with significant segmentation error from the analysis, the revised coefficient of repeatability was reduced to 50 μm (19%). The coefficients of repeatability for all 50 scan set pairs in this study ranged from 34 μm (12%) to 74 μm (32%) for macular subfields A1 to A9. The ACPT and MCPT measures are exquisitely sensitive to eye movement because they only sample the retinal thickness at a single point; consequently, they were found to be the least repeatable measures in these patients with nAMD. The most repeatable measure was TMV, with a coefficient of repeatability of 0.7 mm³ (10%). However, although this suggests that changes in TMV may be more indicative of clinical change, TMV may not be as sensitive in detecting true change as retinal thickness because TMV measurements represent an average over a large area. 

These images were taken from consecutive patients with nAMD and did contain examples of scans with computer algorithm-related retinal boundary placement errors (segmentation error). These computer algorithm errors occurred despite the use of an experienced OCT technician certified by image reading centers for OCT scanning in clinical trials. We accepted the limitations of scan quality and associated errors in retinal boundary placement as illustrating the difficulty of performing OCT imaging in this elderly group of patients with significant macular pathology, poor fixation, and poor visual acuity. 

After excluding scan sets with significant segmentation error from the analysis, the coefficients of repeatability ranged from 33 μm (12%) to 76 μm (28%) for macular subfields A1 to A9, with a coefficient of repeatability of 50 μm (19%) for the central 1-mm (A1) macular subfield. The Bland-Altman plots we constructed allowed us to examine the association between repeatability and retinal thickness. The plots did not reveal any consistent evidence of reduced repeatability with increasing OCT retinal thickness. 

Both routine clinical practice and investigator-determined retreatment decisions in clinical trials for nAMD rely on knowledge of the measurement variability of OCT-determined retinal thickness measurements in nAMD. However, to our knowledge, no studies have specifically addressed this problem; previous studies have concentrated on estimating the intrasession repeatability in healthy persons and patients with diabetic macular edema ^{4 6 8} or intersession variability. ⁹ The study by Massin et al. ⁶ assessed the reproducibility of retinal thickness measurements in 10 eyes of 10 healthy persons and 10 eyes of 10 diabetic patients with clinically significant macular edema (CSME). They found excellent intraobserver reproducibility in healthy persons and diabetic patients, with a repeatability coefficient of less than 7 μm (1.5%) in healthy patients and a repeatability coefficient of less than 27 μm in all ETDRS subfields of the diabetic patients. Similarly, Polito et al. ⁴ reported coefficients of repeatability of approximately 5% for 10 dilated eyes of 10 healthy subjects and 15 dilated eyes of 15 diabetic patients with CSME. Both these studies were conducted using the StratusOCT machine but with earlier versions of the proprietary software. The study by Krzystolik et al. ⁸ for the Diabetic Retinopathy Clinical Research Network analyzed 1205 pairs of OCT scans of patients with CSME and reported a coefficient of repeatability of 38 μm (11%) for the central 1-mm macular subfield. The study by Browning et al. ⁹ estimated the intersession coefficient of repeatability of foveal zone thickness OCT scans in fellow eyes of 19 patients with retinal disease to be approximately 37 μm. However this estimate was made using a small sample size in eyes with no active disease, making it difficult to apply when imaging eyes with nAMD. 

The values of coefficients of repeatability obtained in this study for patients with nAMD appear greater than for diabetic patients with CSME. One reason for this may be the increased incidence of StratusOCT computer algorithm-related errors in mapping the inner and outer retinal boundaries (segmentation error) in patients with nAMD. ^{3 10} Another factor that may contribute to the higher variability in this group of patients may be changes in alignment between the two sets of scans resulting from fixation instability, making it difficult to ensure that the same retinal area is scanned each time. Further studies are required to determine how much these factors contribute to the variability in thickness measurement. Although some studies have advocated repeating OCT scans until retinal boundaries are correctly identified, ² this approach may not be possible in all nAMD patients, with a significant proportion of incorrect computer algorithm detection of retinal boundaries despite the best efforts of the OCT technician. ^{3 10}  

Given that OCT-determined retinal thickness is now used by many physicians as a measure of disease activity and of the need for retreatment with intravitreous agents that target VEGF, ² it is important to attempt to obtain an estimate of repeatability of retinal thickness measurements in these patients to differentiate clinical change from measurement variability. The study by Fung et al. ² used an increase in central macular thickness of at least 100 μm as one of the criteria for retreatment with ranibizumab in patients with nAMD. This does not seem unreasonable given the coefficients of repeatability estimated in our study. Indeed, if scans with significant segmentation error can be excluded, then a change in central macular thickness of approximately 50 μm may be used to guide retreatment in nAMD using our revised estimate of the coefficient of repeatability for the A1 ETDRS macular subfield. Using an experienced observer to manually measure the center-point thickness (CPT) still resulted in large variability of this measure, with a modest reduction in variability when compared with the equivalent automated measure (coefficients of repeatability, 25% for MCPT and 32% for ACPT). This suggests that the CPT alone should not be used to guide retreatment decisions in patients with nAMD. 

It is hoped that the repeatability of retinal thickness measurements in nAMD patients will be improved with the introduction of newer technologies, such as spectral domain OCT ¹¹ and retinal tracking with OCT imaging. ¹² The values obtained in this study could be used to guide retreatment decisions in nAMD patients both in clinical trials, which use investigator-dependent retreatment criteria, and in clinical practice. 

 Submitted for publication September 14, 2007; revised October 8, 2007; accepted January 21, 2008.

 Disclosure: P.J. Patel, None; F.K. Chen, None; F. Ikeji, None; W. Xing, None; C. Bunce, None; L. Da Cruz, None; A. Tufail, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Figure 1.

 

Macular subfields for FMTM protocol analysis. The central A1 field has a diameter of 1 mm, fields A2 to A5 are zones of a circle 3 mm in diameter, and fields A6 to A9 are zones of a circle 6 mm in diameter.

Table 1.

 

Median Retinal Thickness, Interquartile Range, and Wilcoxon Signed Rank Test P Value for Each of the ETDRS Macular Subfields, Automated Center-Point Thickness, Manual Center-Point Thickness, and the Mean Total Macular Volume Measurement for All 50 Patients

Macular Field	Median Retinal Thickness/Volume (interquartile range) (μm)			P (Wilcoxon signed rank test)
		Scan Set 1	Scan Set 2
A1	270 (226–323)	252 (220–323)	0.138
A2	280 (242–332)	276 (242–337)	0.404
A3	270 (242–321)	265 (237–317)	0.746
A4	278 (254–326)	273 (256–321)	0.267
A5	291 (226–348)	293 (248–345)	0.992
A6	231 (216–323)	229 (218–257)	0.299
A7	216 (202–239)	218 (202–250)	0.349
A8	226 (203–263)	225 (202–252)	0.481
A9	253 (239–289)	258 (235–308)	0.599
ACPT	257 (203–329)	243 (211–329)	0.263
MCPT	276 (211–372)	273 (217–361)	0.817
TMV	6.9 (6.5–7.9) mm3	6.9 (6.5–8.0) mm3	0.210

Figure 2.

 

Bland-Altman plot for the central 1 mm (A1) ETDRS macular subfield, with 95% limits of agreement indicated by dotted lines.

Figure 3.

 

Bland-Altman plot for the automated center-point thickness measure, with 95% limits of agreement indicated by dotted lines.

Figure 4.

 

Bland-Altman plot for the manually measured center-point thickness, with 95% limits of agreement indicated by dotted lines.

Table 2.

 

Mean of the Difference between Measurements 1 and 2 with Coefficient of Repeatability (CR) for Each of the ETDRS Subfield ACPT, MCPT, and TMV Measurements

Macular Field	Δ (Scan Set 1 − Scan Set 2) Mean (μm)	CR (1.96 × SD) (μm)	Mean of Two Scan Set Measurements (μm)	CR Expressed as Percentage of Mean*
A1	8	67	287	23
A2	2	50	303	16
A3	5	74	290	26
A4	5	45	299	15
A5	−4	54	313	17
A6	−5	49	248	20
A7	3	56	235	24
A8	5	48	242	20
A9	−2	34	274	12
ACPT	10	88	277	32
MCPT	−1	74	297	25
TMV	0.02 mm3	0.71 mm3	7.4 mm3	10

*  Equates to 1.96 × coefficient of variation.

Table 3.

 

Coefficients of Repeatability (CR) for ETDRS Subfields ACPT, MCPT, and TMV in Scan Sets without Significant Segmentation Error (n = 41)

Macular Field	Δ (Scan Set 1 − Scan Set 2) Mean (μm)	CR (1.96 × SD) (μm)	(Mean of Scan Set 1 + Mean of Scan Set 2)/2 (μm)	CR Expressed as Percentage of Mean*
A1	1	50	268	19
A2	3	48	286	17
A3	5	76	275	28
A4	2	33	282	12
A5	−5	52	293	18
A6	−7	53	239	22
A7	2	45	222	20
A8	3	35	229	15
A9	−2	33	267	12
ACPT	4	66	259	25
MCPT	1	68	260	26
TMV	−0.02 mm3	0.7 mm3	7 mm3	10

*  Equates to 1.96 × coefficient of variation.