Intersession Repeatability of Visual Acuity Scores in Age-Related Macular Degeneration

Abstract

purpose. To describe the intersession repeatability of visual acuity measures obtained with Early Treatment of Diabetic Retinopathy Study (ETDRS) charts in patients with age-related macular degeneration.

methods. Visual acuity was measured in four sessions over 12 weeks using a standardized protocol with ETDRS charts in 107 nontreated eyes of 107 patients with age-related macular degeneration enrolled in an ongoing clinical trial.

results. Data from 90 patients were included in the analysis. The 95% coefficient of repeatability (CR) was 12 ETDRS letters and ranged from 9 letters for 29 eyes with small to intermediate drusen only to 17 letters for 25 eyes with late AMD (macular scars or geographic atrophy). Ten (11%) eyes had a 5-letter reduction or more in visual acuity at the week 1 visit compared with baseline. Excluding seven eyes with visual acuity measurements potentially affected by measurement-related factors (a change in testing distance between visits) the revised CR was 10 letters for the cohort (n = 83) and 11 letters for the late-AMD subgroup (n = 18).

conclusions. Intersession ETDRS visual acuity measurements are subject to considerable variability in patients with AMD. The variability may be due to both measurement- and disease-related factors. The variable readings have implications for AMD clinical trial design and for the assessment and treatment of patients with neovascular AMD. Further work is needed to determine both the sources of variability in visual acuity measurements and the optimal change criterion for visual acuity measurements in this important group of patients.

Distance visual acuity assessment is the most commonly used method to assess visual function in patients with age-related macular degeneration (AMD). Change in distance visual acuity is used as the primary outcome measure in clinical trials for neovascular (n)AMD, ^{1 2 3 4 5} and reduction in visual acuity in combination with other clinical signs of disease activity is one of the criteria used to trigger retreatment with antiangiogenic agents. ⁶ Indeed, reduction in visual acuity of greater than 5 letters is specifically listed on the European Medicines Agency (EMEA) product label for ranibizumab as part of the criteria used to determine retreatment. 

The most widely adopted method of assessing visual acuity in clinical trials is by using Early Treatment of Diabetic Retinopathy Study (ETDRS) charts. These are modified versions of the logarithm of the minimum angle of resolution (logMAR) charts designed by Bailey and Lovie ⁷ and are described in detail elsewhere. ⁸ The merits of the ETDRS charts are well documented ⁸ and have led to their adoption as the gold standard in clinical trials. ⁹ Table 1shows the relationship between the ETDRS letter score and the approximate Snellen visual acuity equivalent. 

To distinguish clinical change from measurement error at successive patient visits both in clinical trials and clinical practice, it is important to establish the repeatability of the clinical method being used. For patients with nAMD, it is difficult to establish the intersession repeatability of visual acuity measurement due to disease progression and prompt treatment. However, we may use the untreated fellow eye of patients receiving active therapy for nAMD to assess the intersession repeatability of ETDRS visual acuity scores in this patient group. Studies have been conducted to assess the repeatability of Bailey-Lovie chart visual acuity scores in a single session in patients with AMD, ¹⁰ the repeatability of ETDRS chart measurements between sessions in patients with a range of advanced eye disease, ¹¹ and the intersession repeatability of Bailey-Lovie chart visual acuity measures in normal elderly patients. ¹² Reported estimates of the 95% coefficient of repeatability of the ETDRS charts ranged from 0.07 logMAR ¹³ (3.5 letters) to 0.16 logMAR ¹⁴ (8 letters) in normal subjects and from 0.15 logMAR ¹⁵ (7.5 letters) to approximately 0.2 to 0.3 logMAR ¹¹ (10 to 15 letters) in patients with reduced vision. This report is the first regarding intersession repeatability of ETDRS visual acuity scores determined with a standardized protocol in a large cohort of patients with AMD. 

Methods

Subjects

Data from the untreated eyes of patients enrolled in the Avastin (bevacizumab) for Choroidal Neovascularization (ABC) Trial (Patel PJ, et al. IOVS 2007;48:ARVO E-Abstract 4536), were used in this repeatability study. This ongoing clinical trial is a prospective, double-masked, randomized, controlled trial investigating the safety and efficacy of intravitreal bevacizumab (Avastin; Genentech, Inc., South San Francisco, CA) in the treatment of nAMD. All patients had consented to visual acuity testing and the research adhered to the tenets of the Declaration of Helsinki. In addition, approval for this research had been obtained both from the ABC Trial Steering Committee and the Research Governance Committee of Moorfields Eye Hospital. Best corrected visual acuity data at baseline, and weeks 1, 6, and 12 from 107 eyes of 107 patients were included. These eyes had a spectrum of disease from drusen to geographic atrophy and macular scars due to nAMD. For analysis purposes, eyes were classified into three AMD subgroups: eyes with early AMD with either small to intermediate (<125 μm) drusen only; eyes with large drusen (>125 μm, with or without pigment changes), and eyes with late AMD (central geographic atrophy or advanced corneal neovascularization [CNV] with subfoveal fibrosis not amenable to further treatment). On clinical examination at the baseline visit, none of these eyes had active, treatable nAMD. Fluorescein angiography and optical coherence tomography (OCT) imaging were used to confirm the absence of clinically detectible disease progression at each visit. The assessment of repeatability of measurements taken over 12 weeks assumes no clinical or subclinical change in disease status over this period causing visual acuity change. Although a reasonable assumption in this cohort, the repeatability of visual acuity scores was also reported for the baseline and week 1 visit only to exclude the effect of subclinical disease progression leading to changes in visual acuity. In addition, the 95% coefficients of repeatability were calculated for different AMD subgroups. Finally, the number of these stable eyes with AMD showing an apparent change in visual acuity at consecutive visits based on the change criteria of 5 or more letters, 10 or more letters, or 15 or more letters was calculated. 

Visual Acuity Measurement

Distance visual acuity measurement was performed by optometrists accredited for clinical trials work in a standardized protocol with ETDRS logMAR charts 1, 2, and R (Lighthouse International, New York, NY). Chart R was used for refraction, with charts 1 and 2 used to test the right or left eyes, respectively, at each visit. Six accredited optometrists performed measurements during the study period. The charts were presented in the light box (Lighthouse International) with a luminance of 150 cd/m² measured with a digital light meter by an independent monitor on two separate visits. A random check of visual acuity measurement and luminance was also performed by trial staff (PJP) to ensure adherence to protocol. Optometrists were masked to previous measurements of visual acuity when undertaking measurements. 

Each patient was refracted at each visit with a standardized protocol. At the baseline visit, the patient’s distance viewing spectacle prescription was measured with a lensometer and used as the beginning approximate refraction. If no spectacles for distance vision were worn, retinoscopy or autorefraction was used as a starting point for refraction. Subjective refraction was then performed refining both the spherical and cylindrical components of the refractive error to give the final correction for both eyes. 

With the appropriate refractive correction in place in the trial frame for the eye being tested and occlusion of the fellow eye, visual acuity testing was performed at a testing distance of 4 m. The patient was asked to read each letter in turn on the chart, guessing where necessary, until four mistakes or more were made on a single line (4 letters or a full row read incorrectly). The ETDRS visual acuity score was calculated by adding 30 to the number of letters correctly read at 4 m. If fewer than 4 letters were read correctly at the initial testing distance of 4 m, testing was repeated at 1 m (adding +0.75 to the spherical prescription). For the patients retested at 1 m, the final ETDRS visual acuity score was calculated by adding the number of letters correctly read at 1 m (up to a maximum of 30 letters) to the number of letters correctly read at 4 m. If a patient was able to read more than 30 letters when retested at 1 m, the patient was stopped at this point to ensure that letters of equivalent visual acuity were not counted twice in the final visual acuity score (when the letters read correctly at 4 m were added to the 1-m score). This procedure was repeated at weeks 1, 6, and 12. 

Statistical Analysis

The mean visual acuity for the entire cohort of patients at each visit was calculated and the difference between means across the four sessions in each of the three disease categories was analyzed by using the mixed within-subject analysis of variance (ANOVA) method. In line with the recommendations outlined by Bland and Altman, ¹⁶ the intrasubject SD of visual acuity scores for individual patients was calculated and plotted against the mean score for each patient (for all four measurements). The mean intrasubject SD (s _w) was used to calculate the 95% coefficient of repeatability (CR), defined by Bland and Altman as 1.96 × √(2s ² _w). The difference between two measurements for the same subject is expected to be less than the CR for 95% of pairs of observations. This calculation was made for the entire cohort of patients with additional values calculated for the three AMD diagnostic categories. To exclude the potential effect of disease progression, separate coefficients of repeatability were calculated for patients using only the baseline and week 1 measurements (a revised plot of individual patients’ SD against mean ETDRS visual acuity letter score for these two measurements is also presented). 

Results

Of the 107 eyes of 107 patients enrolled in a treatment trial whose fellow eyes had active CNV, data from 17 patients were excluded from the analysis: 11 patients were unable to read five letters correctly at 1 m at one or more visits; 2 patients had other ocular disease affecting the visual acuity (corneal scar and amblyopia); 2 patients had active CNV; and 2 patients had incomplete visual acuity data. Complete data from all four measurement points were available in all 90 remaining patients (360 best corrected visual acuity measurements included in the analysis), including 35 male and 55 female patients: 39 left eyes and 51 right eyes. The number of eyes in each AMD category and the mean age of patients are shown in Table 2 . The mean time (±SD) in days between the baseline and week 1 visual acuity measurements was 7 (±3) days. The mean interval between baseline and week 12 visits was 84 (±8) days. Of the 25 patients with late AMD, 5 had central geographic atrophy and 20 had a macular scar due to previous nAMD. 

The mean visual acuity score of patients at each visit is shown in Table 3 . A mixed between-within subject ANOVA was conducted on the 90 subjects. There was no significant interaction between disease category and session (P = 0.55) and variation of visual acuity across the four sessions was not significant (P = 0.23). There was, however, a significant difference in visual acuity across the three disease categories (P < 0.0005), with poorer acuity in the late-AMD group than in the other two groups. 

The CR for patients is shown in Table 4with measurements from patients with small to intermediate drusen being the most repeatable (CR = 9 letters) and measurements from patients with late AMD the least repeatable (CR = 17 letters). The overall CR for this cohort of stable AMD patients for all visits was 12 letters. The plot of intrasubject SD against mean visual acuity score is shown in Figure 1with a trend to increased variability for lower mean visual acuity scores. The CR for only the baseline and week 1 visits was 11 letters. Figure 2shows the revised plot of the intrasubject SD against the mean ETDRS visual acuity score for the data from the baseline and week 1 visits only. 

Seven of these 90 patients (all in the late-AMD subgroup) needed a change of chart distances across visits (between 4- and 1-m testing distances), and this change may have led to less repeatable measurements, due to the method of visual acuity assessment rather than to subject-related factors. After excluding these seven patients were excluded from the analysis, the revised CR for the cohort was 10 letters (n = 83) and 11 letters for the late-AMD subgroup (n = 18). Figure 3shows a revised plot of the intrasubject SD against the mean ETDRS visual acuity for these 83 patients. 

Table 5shows the percentage of this cohort of patients with stable AMD incorrectly classified as improving or decreasing vision using different change criteria between the week 1 and baseline visits (n = 90). In total, 31% of these patients with stable AMD showed an apparent change in vision (11% decreasing and 20% improving vision) using the widely applied change criterion of 1 line (5 letters) or more between baseline and week 1 measurements. 

Discussion

Distance visual acuity measurements are subject to variability. Determining the repeatability of visual acuity measurements is central to the assessment of patients with eye disease. Only by determining the repeatability of these measures can we gain an understanding of whether a change in visual acuity is likely to represent true clinical change rather than measurement error. Determining the reasons for the change is especially important, as reduction in visual acuity is one of the criteria used by investigators to assess the need for retreatment with ranibizumab in patients with nAMD, ⁶ and a defined reduction in vision is stated as part of the retreatment criteria listed on the European product label for this drug. Repeatability of visual acuity measurements in patients with different diseases is also of use when determining endpoints for clinical trials for retinal disease. ¹⁷  

The development of the ETDRS visual acuity chart with standard testing protocols has led to improvements in the repeatability of visual acuity assessment, and this chart is recommended for use in clinical trials. ⁹ Previous studies have estimated the repeatability of visual acuity measurement in patients with age-related macular degeneration in the same visual acuity measurement session. The study by Blackhurst and Maguire ¹⁰ for the Macular Photocoagulation Study (MPS) repeated visual acuity measurements on the same day with two different observers using the Bailey-Lovie chart in 164 eyes of 82 patients with nAMD, ocular histoplasmosis, or idiopathic choroidal neovascularization. In the subgroup with nAMD (60 eyes) the SD of visual acuity scores was 0.93 lines (4.7 letters), similar to our results with an SD of 1 line (5 letters) for the differences in baseline and 1 week visual acuity scores for our subjects. 

Kiser et al. ¹¹ investigated the repeatability of standard visual function tests in patients with severe visual loss. This included 17 patients with macular degeneration (including but not restricted to patients with AMD). These patients were further subdivided into two groups based on visual acuity. The reported CR for this small subgroup of patients was approximately 0.26 log units (13 letters) for the eight patients with better vision (20/200 to 20/500) and approximately 0.22 log units (11 letters) for the eight patients with poorer vision (<20/500). Overall, they found no increase in CR with decreasing visual acuity. We report a CR of 12 letters for this cohort of patients with stable AMD (revised CR of 10 letters after excluding seven patients with visual acuity measurements performed at different testing distances). Although our results suggest a relationship between visual acuity and repeatability, the trend for poorer repeatability with poorer visual acuity seems driven by the increased variability for patients in the 25-to 35-letter visual acuity score range. These patients included the seven patients with visual acuity measurements at different testing distances (4 and 1 m) at different visits. Furthermore, by excluding these seven patients, the late AMD cohort is reduced in number from 25 to 18, with a revised CR of 11 letters which is more consistent with the other subgroups. It is not possible to determine whether the increased variability of visual acuity scores in the seven patients tested at different distances at different visits was entirely due to measurement factors or whether there was contribution from subject or disease-related factors. Further studies are needed to explore which factors contribute to the variability. It is therefore difficult to draw definitive conclusions from our study about the relationship between AMD subgroup, baseline (or mean) visual acuity, and repeatability. 

In Figure 1 , the five patients with the least repeatable (highest SD) visual acuity scores all had measurements at different distances (4 and 1 m) at different visits. It is not surprising that there is increased variability when subjects need to be tested both at 1 and 4 m, as this is effectively the same as performing two visual acuity tests and adding the scores but also adding the variability of each test. Also work from others suggests that visual acuity measurement with ETDRS charts at 1 and 4 m may not be equivalent ¹⁸ leading to less repeatable measurement if patients are tested at different distances at different visits. Furthermore, any such nonequivalence of scores at difference testing distances, ¹⁸ due to the method of scoring ETDRS visual acuity could artificially magnify any gain in visual acuity arising from therapeutic agents as the longer testing distance (4 m in our case or 2 m in other studies) may give better visual acuity scores than shorter testing distances. Patients initially starting at the 1-m distance but who then gain sufficient vision to remain at the 4-m testing distance (4 letters read correctly at 4 m in our protocol but 20 letters read correctly at the longer testing distance in other protocols) may therefore gain additional letters due to the scoring method that gives better visual acuity at longer testing distances. This effect may have led to an overestimate of the proportion of patients gaining 15 ETDRS letters or more in the pivotal trials for the treatment of exudative AMD. ^{4 5}  

An alternative method of scoring visual acuity measurements of patients tested initially at 4 m and then at 1 m is to count the letters read at 1 m rather than adding the number of letters read at 4 m to the score at 1 m. Further study is needed to see whether this approach improves the repeatability of visual acuity measurements in patients with visual acuity scores requiring assessment at different testing distances at different visits. A reanalysis of our results was not possible, as the patients were not encouraged to continue reading after 30 letters had been correctly identified at 1 m, preventing a complete assessment of visual acuity at this testing distance. The scoring strategy of combining scores from the two testing distances adopted in this study reflects practice in most clinical trials; however, different testing distances may be used in other studies with different criteria in determining when to move the patient to the 1-m testing distance (e.g., fewer than 20 letters read correctly at the longer testing distance). 

Other sources of intersession measurement variability include variability or change in subjective refraction, variability in measurement method, change in disease state, and patient-related factors. As in all clinical trials, we attempted to minimize variability in measurement method by using standardized refraction and scoring protocols and ensuring strict adherence to these protocols. To address the potential effect of subclinical disease progression on visual acuity scores over an interval of 12 weeks, we measured the repeatability, both for the four visits over this period and for the two initial measurements separated by 1 week. The CRs are similar for these two test–retest intervals, suggesting little disease progression over the 12-week period. In addition, patients with evidence of disease progression (determined by clinical examination, fluorescein angiography or OCT imaging) were excluded from the analysis. Patient-related factors leading to variability in measurement include changes in fixation (variable or multiple preferred retinal loci). The existence of multiple and variable preferred retinal loci for fixation in patients with AMD ¹⁹ may underlie the apparent changes in visual acuity seen at different visits in the absence of AMD disease progression and may explain the degree of visual acuity test–retest variability in some patients. 

In this cohort of 90 patients with AMD, it is interesting to note that 28 patients had a change in visual acuity of 5 or more ETDRS letters when retested 1 week after the initial assessment with 10 patients showing an apparent decrease in vision (Table 5) . Applying this to patients with nAMD undergoing ranibizumab treatment with standardized retreatment strategies, ⁶ approximately 1 in 10 patients with stable disease may show a reduction of 5 letters or more in ETDRS visual acuity letter scores at consecutive visits through test–retest variability alone. The arbitrary criterion of loss of greater than 5 letters in visual acuity alone as evidence of disease progression cited in the EMEA product label for ranibizumab is therefore problematic. Our results suggest that the specificity for disease progression could be substantially increased by applying a change criterion of 10 letters; however, this would reduce sensitivity in detecting disease progression. Further work is needed to explore the optimal change criterion for visual acuity measurement in AMD. 

In view of the large variability of ETDRS visual acuity measurements in patients with AMD, clinical trials may benefit from using computerized methods of visual acuity measurement methods which have been shown to improve the repeatability of measurements in normal subjects. ²⁰  

The advantages of this study include the large sample size and the use of a standardized protocol in a clinical trial setting, to minimize interobserver variability. One limitation is that the study was not designed specifically as a repeatability study but used visual acuity data from an ongoing clinical trial. However, the use of these data may also be viewed as a strength, as although studies specifically designed to measure repeatability may achieve an extremely high degree of repeatability when using researchers dedicated to this goal, the values may not be easily applied to other settings. The optometrists in this study were performing measurements adhering to a standardized clinical trial protocol but without the knowledge that the repeatability of measurements would be formally assessed. This better reflects a true clinical trial setting in which many measurements are taken by observers after set protocols but often without formal assessment of intersession repeatability. 

In summary, this is the first study to report the intersession repeatability of ETDRS visual acuity scores in a large cohort of patients with AMD in a clinical trials setting. The values obtained in this study may be used to guide retreatment strategies as well as future clinical trial design and provide an insight into the repeatability of ETDRS visual acuity measurement in this important group of patients. 

 Submitted for publication February 26, 2008; revised April 24 and June 9, 2008; accepted August 22, 2008.

 Disclosure: P.J. Patel, None; F.K. Chen, None; G. S. Rubin, None; A. Tufai l, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Table 1.

 

The Relationship between the ETDRS Visual Acuity Score and the Approximate Snellen Visual Acuity

ETDRS Letter Score*	Approximate Snellen Visual Acuity Equivalent
		In Feet	In Meters
25	20/317	6/95
30	20/252	6/76
35	20/200	6/60
40	20/159	6/48
45	20/126	6/38
50	20/100	6/30
55	20/80	6/24
60	20/63	6/19
65	20/50	6/15
70	20/40	6/12
75	20/32	6/10
80	20/25	6/8
85	20/20	6/6

*  Total number of letters read correctly at 1 meter. For patients tested at 4 m, add 30 letters to the number of letters read correctly at 4 m to obtain the ETDRS letter score (add 15 letters for patients tested at 2 m).

Table 2.

 

Diagnoses and Age of Patients

	Diagnosis	Eyes (n)	Mean Age in Years (SD)	Median Age in Years (Range)
Early AMD	Small/Intermediate drusen	29	77 (7)	76 (62–87)
Early AMD	Large drusen with or without pigment change	36	79 (7)	80 (62–88)
Late AMD		25	80 (7)	81 (65–92)
Total		90	78 (7)	80 (62–92)

Table 3.

 

Mean ETDRS Visual Acuity Scores at Each Visit

	Diagnosis	Eyes (n)	Baseline	Week 1	Week 6	Week 12
Early AMD	Small/Intermediate drusen	29	80 (11)	80 (14)	81 (15)	79 (13)
Early AMD	Large drusen with or without pigment change	36	75 (9)	76 (9)	76 (8)	76 (9)
Late AMD		25	37 (20)	38 (21)	40 (21)	39 (22)
Total		90	66 (23)	67 (23)	68 (23)	67 (23)

 Data are mean scores in letters (SD).

Table 4.

 

Mean CR for Visual Acuity Scores by Diagnosis

	Diagnosis	All Visits (n)	Baseline and Week 1 Visits Only (n)
Early AMD	Small/intermediate drusen	9 (29)	9 (29)
Early AMD	Large drusen with or without pigment change	10 (36)	8 (36)
Late AMD		17 (25)*	15 (25)
Total		12 (90)*	11 (90)

 Data are mean CR (ETDRS letters).

*  Revised CR = 11 letters for the late AMD subgroup (n = 18) and 10 letters for entire cohort (n = 83) when seven patients with visual acuity scores assessed at different testing distances at different visits are excluded.

Figure 1.

 

Plot of individual patients’ standard deviation against mean visual acuity score for all four measurement visits (n = 90).

Figure 2.

 

Plot of individual patients’ standard deviation against mean visual acuity score for the baseline and week 1 measurement visits only (n = 90).

Figure 3.

 

Plot of individual patients’ standard deviation against mean visual acuity score for all four measurement visits, excluding patients with visual acuity scores assessed at different testing distances at different visits (n = 83).

Table 5.

 

Number of Clinically Stable AMD Patients Showing Apparent Change in Visual Acuity between Week 1 and Baseline Visits Using Different Change Criteria

Change Criterion (ETDRS Letters)	Clinically Stable Patients Showing Apparent Visual Acuity Change at Week 1 Visit Compared to Baseline n (%)
		Decreased Visual Acuity	Improved Visual Acuity
≥5	10 (11)	18 (20)
≥10	2 (2)	3 (3)
≥15	1 (1)	1 (1)