Effect of Scoring and Termination Rules on Test–Retest Variability of a Novel High-Pass Letter Acuity Chart

Abstract

Purpose.: Test–retest variability (TRV) limits our ability to detect clinically significant changes in visual acuity (VA). We wanted to compare the effect of scoring and termination rules on TRV for logMAR charts, employing either conventional or pseudo high-pass (Vanishing Optotype) letters.

Methods.: VA measurements and TRV were compared in 50 uncorrected normal observers (17 male, mean age 42.8 ± 16.2 years) using both conventional logMAR-style charts and letter charts of the same layout but containing pseudo high-pass letters (Moorfields Acuity Chart [MAC]). Additional charts employing a different 10-letter alphabet to the Sloan set were also tested. Mean spherical refractive error was −0.93 diopters (D; range, −5.38 to +3.00 D). Acuity scores were calculated using three methods: letter-by-letter, with either line- or chart-based termination, and line-by-line scoring. Bland–Altman methods were used to calculate 95% ranges for TRV.

Results.: While acuity thresholds were higher for the MAC, they were less affected by termination criteria and displayed significantly lower 95% TRV values across all scoring techniques. Ordinary least squares regression analysis confirmed a proportional as well as systematic bias between conventional and MAC measurements (r ² = 0.217, P = 0.001) such that the difference between the two was greater with better VA.

Conclusions.: TRV was consistently lower for a logMAR chart employing high-pass rather than conventional letters in uncorrected refractive error and was less affected by termination and scoring methods. The MAC was also less affected by optical defocus. Further work is required to determine the usefulness of different charts to differentiate between optical and neural losses of vision.

Introduction

Measurement of visual acuity is the most widely used test in ophthalmic care, and the single most important way of characterizing visual function when detecting or monitoring refractive error or ocular disease. In addition, it is the standard outcome measure of treatment efficacy in many clinical trials. ^1–5  

A long-standing weakness of conventional measures of visual acuity is their test–retest variability (TRV). TRV quantifies the variability in test results when the test is repeatedly administered to the same individual, even in the absence of true clinical change. In order that a change in clinical status should not be masked by measurement error, tests of visual acuity must not only be valid but also reliable, the latter being quantified by TRV. In clinical measurements, this is commonly expressed as a Bland–Altman 95% limit of agreement. 

The Snellen chart has high (i.e., poor) TRV with values ranging from ±5 to 16.5 letters in normal subjects ⁶ and up to 0.33 logMAR in a group of subjects with cataracts, pseudophakia, or early-stage glaucoma. ⁷ TRV is known to be further influenced by factors such as the chart design, ^6,8,9 optical defocus, ^10,11 the ocular status of the observer, ^12,13 intra-examiner variability, ¹⁴ the number of alternative letter choices given to the observer, ^15,16 and the scoring and test termination rules employed. ^16,17 The large TRV associated with the Snellen chart has often been attributed to the line scoring method employed, in which the visual acuity score is the smallest row in which a specified proportion of letters are read correctly. LogMAR charts such as the ETDRS and Bailey–Lovie charts, were designed to overcome some of the limitations of Snellen charts and are now employed as the “gold standard.” Vanden Bosch and Wall ¹⁸ showed that a single-letter scoring algorithm, in which a score is given to each letter read, reduces TRV. Using this method, Laidlaw et al. ¹⁹ demonstrated a reduction in TRV from 0.20 to 0.14 logMAR; Bailey and Lovie, ⁹ a change from 2 to 1 logMAR lines; and Arditi and Cagenello, ¹⁷ a change from 0.13 to 0.09 logMAR using the ETDRS chart. Applying this technique to Snellen charts is complicated by virtue of the unequal number of letters per line and the unequal jumps between lines, but a score for each letter can still be calculated. 

Another factor potentially affecting acuity TRV is the termination criterion. Carkeet ¹⁶ combined exact calculation and Monte Carlo simulation to investigate the effect of termination rules on TRV. He found that Arditi and Cagenello's ¹⁷ recommendation that subjects attempt all symbols on a chart should, in theory, give the lowest variance only if there is an infinite number of alternative letter choices. Carkeet ¹⁶ investigated the effect of using termination rules in which a set number of letters are read incorrectly on a line (a line-by-line termination rule) and the effect of using a set number of mistakes across the whole chart (a letter-by-letter termination rule). He suggested that, for the ETDRS chart and single-letter scoring, termination rules of four mistakes or more on a line should be used. 

Although logMAR visual acuity charts have corrected many of the deficiencies ascribed to the Snellen chart, TRV nevertheless remains high using these charts. Even with letter-by-letter scoring, values of up to 2 logMAR lines have been reported, ^{8,17,18,20–26} and this value increases in the presence of optical defocus ¹⁰ or retinal disease. ¹³  

The 10 Sloan letters that are used in the ETDRS chart are chosen in combinations so as to give each line the same average difficulty. ²⁷ However, if a test chart's within-line legibility differences are greater than its between-line legibility differences, the test may display higher variability. ²⁸ While by-letter scoring may help to mitigate this, our previous study, ²⁹ which examined the individual recognition thresholds of letters, suggested that it may be possible to select 10 different letters that have much more individually similar recognition thresholds that could further reduce TRV. 

Another potential way to reduce TRV is to employ a different letter design. “Vanishing Optotypes,” first described by Howland et al., ³⁰ have a pseudo high-pass construction consisting of a black “core” with white edges (or vice versa) with the letter being presented on a gray background. Letters are constructed so that their mean-luminance is equal to the background, resulting in thresholds for detection and recognition being almost identical (under foveal viewing), and to the illusion that the letters have “vanished” soon after their discrimination threshold has been reached. One possible advantage of using such letters to measure visual acuity is that previous studies have shown that conventional letters differ significantly in their low spatial frequency content. ^31–33 If the low frequencies are removed, then letters may become more similar, more closely equating their legibility and so reducing the TRV. Initial work by our group ^15,29 supports this notion and has demonstrated that the effect of the number of alternatives available and the actual letter identity on the visual acuity score and variability is greatly reduced for high-pass letters compared to conventional letters. 

Here we investigated the effects of different test termination criteria and scoring methods on estimates of visual acuity and associated TRV, using charts composed of either conventional or high-pass letters in normal subjects with differing levels of uncorrected refractive error. Additionally, we compared the performance (in terms of threshold and TRV) of charts using the set of 10 Sloan letters to identically constructed charts employing an alternative alphabet set chosen from a previous study ²⁹ to contain no “circular” letters (which were found to behave as a separate subset) and be more equally legible. 

This study should allow us to determine the vulnerability of the different chart designs to refractive losses of vision, thus permitting future comparison with performance in neural visual loss, in order to better adopt appropriate tests to discriminate neural and optical losses of vision. 

Methods

Ethical approval for this study was obtained from the UCL Research Ethics Committee and all procedures adhered to the tenets of the Declaration of Helsinki. 

Eight different charts were constructed. The first two, termed Conventional Sloan letter set 1 and 2 (CS1 and CS2), were almost identical in format to ETDRS charts 1 and 2; that is, they used the 10 Sloan letters (C D H K N O R S V Z) of conventional 5 × 5 matrix black-on-white letter design. Letters on a line were spaced a letter width apart and each line was spaced a letter height from the line above. The third and fourth charts, termed the Moorfields Acuity Chart Sloan letter set 1 and 2 (MAC S1 and MAC S2) were of identical construction except that they employed a high-pass letter design. The visual acuity range on the charts (based on stroke width) was 1.20 to −0.20 logMAR. The letter sizes of all charts commenced at 1.2 logMAR instead of 1.0. The range was extended at the poorer acuity end in order to avoid having to use different test distances between charts, since our previous studies had demonstrated overall “poorer” logMAR visual acuities with high-pass compared to conventional letters. 

The fifth and sixth charts had the same design as CS1 and CS2 but used a different 10-letter alphabet (B E H K N P R S X Z) whose legibility was more similar in conventional form. ²⁹ These charts were referred to as Conventional New letter set 1 and 2 (CN1 and CN2). The seventh and eighth charts, the Moorfields Acuity Chart New letter set 1 and 2 (MAC N1 and MAC N2) used this same alphabet in high-pass form. The appearance of charts CS1, MAC S1, CN1, and MAC N1 is shown in Figure 1. 

Figure 1

 

Appearance of charts CS1 and MAC S1 (top), which use the Sloan alphabet, and CN1 and MAC N1 (bottom), which use the New alphabet.

Fifty normal observers were recruited from a primary care optometric practice and from the staff of Moorfields Eye Hospital. Ages ranged from 20 to 76 years (mean age 42.8 years) and 17 participants were male. The right eye was tested in half of the subjects (25) and the left eye in the other half. Each observer underwent preliminary tests to screen for their eligibility to participate in the study. These tests included baseline refraction (retinoscopy and subjective) at 4 m and ocular examination (including binocular indirect ophthalmoscopy) to exclude any ocular pathology. The mean spherical refractive error was −0.93 diopters (D; range, −5.38 to +3.00 D). If, on initial screening, a potential subject's acuity fell outside the measurement range of the charts at 4 m they were excluded, with the result that the mean CS1 unaided letter acuity for the 50 eligible subjects was 0.44 logMAR (range, −0.04 to 1.16 logMAR). 

Unaided acuity was then measured on each subject using each of the eight different charts in one single visit. Subjects were required to read from the top of each chart and encouraged to guess if they were unsure of a letter's identity. Viewing time was not restricted. The test was terminated when a whole line was guessed incorrectly and responses for each letter recorded by the examiner on a pro forma data sheet. The eye to be tested was randomized, as was the order in which the test charts were administered, to control for both learning and fatigue effects. 

Visual acuity scores were then determined in three different ways ^16,18 : 

•  
  Letter-by-letter scoring with line-based termination in which each letter was assigned a value of 0.02 logMAR and the test was terminated when a specified number of letters per line was read incorrectly (i.e., five, four or more, three or more, two or more, or one or more wrong per line). If, for example, in the case of one or more letters being read on a line, the full line being attempted was included in the score, and acuity was recorded as 1.3 − (0.02 × number of correct letters).
•  
  Letter-by-letter scoring with whole-chart termination in which each letter was assigned a score of 0.02 logMAR, but test termination was taken as a specified number of letters wrong across the whole chart (i.e., five or more, four or more, three or more, two or more, one or more letters wrong across the chart). Acuity was again recorded as 1.3 − (0.02 × number of correct letters).
•  
  Line by line scoring, where the visual acuity score was taken as the last line in which more letters were read correctly than incorrectly.

The first two of these techniques employ the same scoring method but differ in their termination criteria. The third technique has a different scoring method. 

Statistical Analysis

For each individual, the difference in the two acuity measurements made with each chart was calculated for each scoring and test termination rule, and the methods of Bland and Altman ³⁴ and ordinary least squares regression analysis were employed. The Shapiro–Wilk W-test was used to confirm that the differences were normally distributed. The GraphPad Prism statistical analysis package (GraphPad Software, Inc., La Jolla, CA) was employed for this purpose. 

Results

Bland and Altman scatter plots were constructed to graphically present the spread of results for the CS1 and CS2, MAC S1 and MAC S2, CN1 and CN2, and finally MAC N1 and MAC N2 charts (Figs. 2a–d). These plots show the TRV of each measurement technique (here, 1.96 × SD) alongside the mean difference. The data presented here are for the letter-by-letter scoring line-based termination for five letters wrong. There was no evidence of any systematic association between the level of agreement and the underlying acuity. The differences were assumed normally distributed using the Shapiro–Wilk W-test (P = 0.251 for CS1 and CS2, P = 0.309 for MAC S1 and MAC S2, P = 0.412 for CN1 and CN2, P = 0.286 for MAC N1 and MAC N2). On this basis we are justified in using Bland and Altman summary statistics of mean bias and 95% limits of agreement, which are presented in the Table. 

Figure 2

 

Bland–Altman plot for test and retest measurements using letter-by-letter line-based termination for five letters wrong for (a) CS1 versus CS2, (b) MAC S1 versus MAC S2, (c) CN1 versus CN2, (d) MAC N1 versus MAC N2, and (e) CS1 versus MAC S1. In each instance (a–d), the mean difference and 95% limits of agreement are plotted and for (e) the slope fitted using ordinary least squares regression analysis r ² = 0.217, P = 0.001.

Table

 

Bland–Altman Summary Statistics of Mean Bias and TRV Expressed as 95% Limits of Agreement

	Mean Difference (SE)	95% CI Mean Difference	Range of Observed Differences	TRV, 95% Limits of Agreement
CS1–CS2	−0.001 (0.010)	−0.020, 0.019	−0.18, 0.12	±0.135
MAC S1–MAC S2	−0.004 (0.007)	−0.017, 0.010	−0.10, 0.12	±0.095
CN1–CN2	0.008 (0.010)	−0.012, 0.029	−0.18, 0.16	±0.142
MAC N1–MAC N2	0.022 (0.008)	0.007, 0.037	−0.08, 0.14	±0.105

 These data are presented graphically in Figure 2a through 2d and visual acuity scores were calculated using letter-by-letter line-based termination for five letters wrong.

The F-test was used to test the null hypothesis that the standard deviation for CS1 is equal to that of MAC S1 and this was rejected (one-tailed P = 0.05, F _49,49 = 1.61). No difference was found between the TRVs for the Sloan letter set and the new alphabet set in either conventional (±0.14 logMAR for both) or high-pass (±0.10 logMAR for both) format. 

Figure 2e displays the comparison results for charts CS1 and MAC S1 for a letter-by-letter line-based termination for five letters wrong criterion. A proportional as well as a systematic bias can be inferred from this in that a greater level of disagreement between the two chart types existed at the better acuity end. The methods described by Bland and Altman do not further inform about such a potential proportional bias, so we performed ordinary least squares regression analysis on the data, confirming the presence of the bias (r ² = 0.217, P = 0.001). The difference in visual acuity between CS1 and MAC S1 was approximately −0.20 logMAR at the 0.00 logMAR visual acuity level (indicating that conventional visual acuity measurements were, on average, 2 logMAR lines “better” than high-pass acuity measurements), compared to a difference of only −0.05 logMAR (half a line) at the 1.00 logMAR acuity level. 

Figures 3a and 3b demonstrate how TRV for letter-by-letter scoring changed with different line and chart termination rules, respectively. Looking at Figure 3a, it can be seen that the TRV for CS1–CS2 was ±0.14 logMAR for five and four or more letters wrong per line termination criteria but increased (to ±0.17 logMAR) for three or more letters wrong. For MAC S1–MAC S2, the TRV was lower and remained at ±0.10 logMAR for five, four or more, and three or more letters wrong per line, increasing (to ±0.13 logMAR) for two or more letters wrong per line. The New alphabet set yielded no systematic improvement in TRV in either CN1 or MAC N1 form under line-based termination criteria. 

Figure 3

 

TRV for letter-by-letter scoring and (a) line- and (b) chart-based termination with different numbers of letters wrong.

For chart-based termination (Fig. 3b), TRV for CS1–CS2 was even higher than with line-based termination with a value of ±0.17 logMAR for five or more errors and four or more errors across the chart, increasing slowly thereafter. For MAC S1–MAC S2 and chart-based termination, the TRV was ±0.10 logMAR (similar to that for line-based termination) with five or more letters wrong across the whole chart, increasing slowly thereafter by around half a letter with four or more errors, and by half a letter again for three or more errors across the chart. Once again, no systematic improvement was found in TRV with the MAC N1 compared to MAC S1 chart with chart based termination rules. A small improvement of one letter was found with the CN1 compared to CS1 chart for five, four, and three or more errors across the chart. 

For line-by-line scoring (not plotted) TRV was 0.23 logMAR for both CS1–CS2 and CN1–CN2, and much lower (0.14 logMAR) for both MAC S1–MAC S2 and MAC N1–MAC N2. 

Figures 4a and 4b demonstrate how the mean logMAR visual acuity (based on stroke width) changed with letter-by-letter scoring for line and chart-based termination rules, respectively. Figure 4a shows how the mean visual acuity score for CS1 with line-based termination was 0.44 logMAR for five letters wrong per line, increasing to 0.45 logMAR with four or more letters wrong, and 0.46 logMAR with three or more letters wrong. The visual acuity threshold then increased to a value of 0.49 logMAR for a termination rule of two or more letters wrong per line. As previously mentioned, the mean logMAR score was larger with MAC S1 (0.57 logMAR) than CS1 for five letters wrong per line but little change in acuity was observed until a termination criterion of one or more letters wrong per line was reached, when threshold rose to 0.63 logMAR. 

Figure 4

 

Mean visual acuity for letter-by-letter scoring with (a) line- and (b) chart-based termination with different number of letters wrong.

Chart-based termination (Fig. 4b) showed a similar pattern, with a larger mean threshold score for MAC S1 (0.58 logMAR) than for CS1 (0.46 logMAR) for five or more letters wrong across the whole chart. CS1 visual acuity thresholds increased by around half a letter with each criterion change until three or more letters wrong across the whole chart, rising to 0.51 logMAR for two or more letters wrong and 0.55 for one or more letters wrong. Once again, little change was noticed in thresholds with the MAC chart until a termination criterion of one or more letters wrong across the chart when the threshold was 0.63 logMAR. 

For line-by-line scoring (not plotted), mean visual acuity was 0.44 logMAR for CS1, 0.41 logMAR for CN1, 0.56 logMAR for MAC S1, and 0.59 for MAC N1. 

Discussion

TRV fundamentally limits the smallest detectable change in visual acuity that can reliably be attributed to a change in clinical status rather than just measurement error. Previous studies have concluded that logMAR charts are superior to Snellen charts since they achieve lower TRV by using improved test design features and scoring techniques. ^9,27 Although Carkeet ¹⁶ has investigated the effect of different termination rules on visual acuity using logMAR charts and single-letter scoring, that work was based on simulation results, whereas we set out to explore these issues by measuring human performance. Furthermore, our previous work with high-pass letters ^15,29 suggested that it may be appropriate to create a chart using these letters (the Moorfields Acuity Chart) to further reduce TRV. 

We found that, regardless of scoring technique (letter-by-letter or line-by-line) or termination criteria (line-based or chart-based), the TRV values for the Moorfields Acuity Charts, employing high-pass letters, are lower than for logMAR charts, composed of conventional letters, in uncorrected normal subjects. Furthermore, while the conventional letter charts were found to have higher TRV values using chart-based (0.17 logMAR at best) as opposed to line-based termination (0.14 logMAR at best) with letter-by-letter scoring, and higher still using line-by-line scoring techniques (0.23 logMAR), the high-pass charts were found to have more similar values across all three methods. In addition, TRV for high-pass charts appears to be less affected by different letter-by-letter termination number criteria compared to the conventional letter charts. While some of the differences may seem modest (0.10 vs. 0.14 logMAR at five errors on a line), they are still of similar magnitude to the improvements in TRV reported by previous studies comparing logMAR charts with Snellen charts ³⁵ or different termination criteria. ^16,17,19  

These observations apply for both the Sloan alphabet and the New alternative alphabet, which excludes circular letters. The 10 Sloan letters used on the ETDRS chart are arranged in combinations such that each line has the same average difficulty. ²⁷ Over the years, it has been commonly thought that using a set of letters with more similar legibility would improve the repeatability of acuity measurements. ⁹ Arditi and Cagenello ¹⁷ investigated the effect on reliability of randomly arranging the 10 Sloan letters and found no difference between the two. Raasch et al. ³⁶ selected a new letter set in which the probability of identification curves was supposedly identical for all the letters and compared the variability scores attained with these to those for the Sloan letters. They found minimal difference between the two, suggesting that the difference in identifiability of the Sloan letters is not a significant factor. In agreement with this, we found minimal differences in TRV between the Sloan set of 10 letters and our New alternative alphabet, indicating that the improvement mostly results from using the high-pass letter design rather than a different alphabet. TRV has previously been shown to increase with optical defocus in conventional letter charts, and it may be that the lower TRV found with the high-pass letters is a result of their greater resistance to blur. ²⁹  

The data in Figure 3a support Carkeet's ¹⁶ recommendation of using a line-by-line termination criterion of four or five letters wrong per line with the ETDRS chart because these gave a TRV value of ±0.14 logMAR, increasing to ±0.17 logMAR with a termination of three or more letters wrong. The results for the conventional letter design charts are typical of those published in other studies of TRV for logMAR acuity measurements in which TRV values of up to 0.2 logMAR can be observed in subjects with unchanged acuity. ^{8,17,18,20–26}  

In assessing the performance of the different chart designs in this study, we have directly compared the logMAR values based on “stroke-width,” by which we mean the total (outside edge to outside edge) angular value of the limbs of the letters in question. The high-pass letters are, of course, constructed of black and white lines that are considerably narrower than the overall stroke-width, meaning that the visual system must rely on finer features (in the spatial domain) or higher letter frequencies (in the frequency domain) and the letter must be made larger to be correctly resolved. It is hardly surprising, therefore, that a systematic bias was observed between CS1 and MAC S1 measurements in that acuity for the high-pass letters appeared worse overall than for conventional letters. Previous studies have shown that the acuity value, if based on stroke or gap width, can vary considerably depending on the design of the letter chart, ^6,37 the number of alternatives, ^15,16 the targets employed to measure it (e.g., gratings, ³¹ Landolt rings, ^38,39 vernier⁴⁰), and even the choice of letter subset ^15,41 depending on the features or spatial frequencies made available to the visual system. In fact, it is increasingly clear that conventional letters should not be regarded as a set of uniformly discriminable stimuli, but really as a group of stimuli, all of very different construction, for which the visual system uses different spatial frequencies to recognize different letters. ^31,33,41,42  

The pattern of the points in the Bland–Altman plot in Figure 2e also suggests that the magnitude of the difference between measurements decreases with worsening acuity. Such a substantial proportional bias is not unusual in acuity chart comparisons and has also been previously demonstrated when comparing Snellen and ETDRS charts. ⁴³ In a Snellen versus ETDRS comparison the reason for the bias must lie in the design of the chart (since the letter design is the same). In the current study, the reason must lie in the design of the letters since the chart layouts were identical. Our previous study ²⁹ found that high-pass letters were more robust to induced optical defocus than conventional letters in that acuity for conventional letters was “better” with zero blur, was roughly equal at 1 D blur, but became “worse” at 2 D blur and beyond. The proportional bias observed in the current study is in agreement with this previous finding in that, as acuity owing to uncorrected refractive error declines, the acuity with the high-pass chart was affected less. This means that those with good visual acuity as a result of small refractive error, will achieve a score of approximately −0.20 logMAR lines better with conventional compared to high-pass letters. Those with poorer acuity resulting from uncorrected refractive error will attain a score of only −0.05 logMAR better. 

In conclusion, it would appear that the new Moorfields Acuity Chart, which employs high-pass letters, displays a lower TRV with uncorrected refractive error than an ETDRS style chart employing conventional letters. It also seems to be somewhat less affected by different termination criteria and scoring methods. Further work is required to test the chart's performance in conditions resulting in neural rather than optical acuity loss to determine its ability to separate these two factors in patients suffering from diseases such as age-related macular degeneration. It may be that different charts, employing letter designs that are differently vulnerable to optical or neural losses of vision, could assist in differentiating between the two. 

Acknowledgments

Supported by a Fight for Sight studentship and by the Special Trustees of Moorfields Eye Hospital. 

Disclosure: N. Shah, None; S.C. Dakin, None; H.L. Whitaker, None; R.S. Anderson, None