Validating Rules for Defining No Improvement of Visual Acuity in Childhood Amblyopia

Abstract

Purpose: When treating amblyopia, it is important to define when visual acuity (VA) is no longer improving (i.e., stable) because treatment decisions may be altered based on this determination.

Methods: Simulated observed VAs, incorporating measurement error, were compared with simulated true VAs to determine false-positive and false-negative rates for stable VA for six rules (using single VA or test/retest measurements, with or without averaging, over two or three visits). Four HOTV VA profiles were modeled: stable or improving VA over time with each of patching and spectacles.

Results: Across six rules and two treatments, when true VA was stable, false-negative rates for stability ranged from 26% to 67%; when true VA was improving, false-positive rates for stability ranged from 0% to 38%. Single VA measurements at consecutive visits had a false-negative rate of 30% with patching and 29% with spectacles, a false-positive rate of 38% with patching and 35% with spectacles. Averaging two VA tests at each visit slightly increased the false-negative rate (35% with patching and 36% with spectacles), while reducing the false-positive rate (22% with patching and 21% with spectacles).

Conclusions: Comparing false-negative and false-positive rates for stability across rules allows selection of the most appropriate rule for clinical practice or research. When considering less desirable treatments, a rule with a lower false-negative rate is preferable, whereas a rule with a lower false-positive rate would be preferred when it is important to correctly classify improving VA.

When treating patients with amblyopia or managing subjects in a research treatment protocol, decisions are made when to start, stop, or change a treatment based on visual acuity (VA). Typically, clinicians and researchers use guidelines or rules to define when VA is no longer improving. In clinical practice, comparing a single VA assessment across two consecutive visits is often used to make this determination, whereas in research studies, other rules may be used. 

The limitations of defining “no improvement” based on single VA measurements several weeks apart have been previously reported.¹ In a study conducted by the Pediatric Eye Disease Investigator Group (PEDIG), children four to less than seven years old with anisometropic, strabismic, or combined-mechanism amblyopia,¹ who had not been wearing their optimum refractive correction for 16 weeks before enrollment, were required to demonstrate stability of amblyopic eye VA with optimum refractive correction before enrollment. Stability was defined as <0.1 logMAR improvement in VA between two consecutive visits eight weeks apart. At the four-week primary outcome visit, those in the spectacle group had a mean VA improvement of 0.06 logMAR with 54% having improved 0.1 logMAR or more and 19% improved 0.2 or more logMAR.¹ These continued improvements in amblyopic eye VA indicate that VA may not have been completely stable at the time when “no improvement” was declared. Although there are other reasons why mean VA might improve, such as regression to the mean, these data suggest that the rule for defining no improvement as 0.1 logMAR or less based on a single measure of VA at two consecutive assessments at least eight weeks apart may have been inadequate in defining stability and suggests that alternative rules should be explored. Simulations provide a tool to compare alternative rules for defining no improvement in VA by generating a very large data set approximating real-world data and allowing exploration of the influence of specific parameters through sensitivity analyses. 

The purpose of the present study was to use simulations based on data from previous PEDIG studies^2–7 to determine the false-positive rates for stability (declaring stability when there was, in fact, improvement) and false-negative rates for stability (declaring improvement when, in fact, there was stability) for six candidate rules. We chose to simulate rules for no improvement and refer to it as “stability” throughout the manuscript. Two common treatments for amblyopia (patching and spectacles) were simulated, which, like most treatments, are not expected to result in worse VA. The rules for classifying stability in VA differ in the number of VA tests per visit (single test, better of test and retest, or average of test and retest) and the number of consecutive visits across which the VAs are compared. Comparing the false-positive rate and the false-negative rate for stability for each rule allows clinicians or researchers to select the most appropriate rule based on the risks of continuing, stopping, or changing treatment or study design. 

Methods

Simulation of Visual Acuities

Four hypothetical profiles of true VA were simulated: (1) improving VA in children being treated for amblyopia with patching for 56 weeks; (2) stable VA with the same baseline VA from no. 1 repeated over 56 weeks; (3) improving VA in children being treated for amblyopia with spectacles for 56 weeks; and (4) stable VA with the same baseline VA from no. 3 repeated over 56 weeks. The simulation procedure is described in detail in Supplementary Appendix A1. 

Baseline and follow-up VAs in children three to less than seven years old prescribed patching^2–5 or prescribed spectacle treatment⁶ from previous PEDIG studies, measured using the Amblyopia Treatment Study HOTV protocol⁸ in logarithm of the minimum angle of resolution (logMAR) increments, were used to simulate the corresponding true VA profiles. The baseline VAs of these study subjects were used to estimate two different Beta distributions for the patching and spectacles cohorts, respectively, from which the simulated baseline VAs were randomly drawn. Each set of simulations had 10,000 hypothetical subjects, and each subject had a “true” baseline VA randomly sampled from the Beta distribution. The improving “true” VAs over time were derived from a linear mixed model of follow-up VAs from previous studies,^2–6 generating profiles similar to an exponentially decreasing improvement of VA. 

For each simulated “true” VA, two “observed” VAs (representing an initial test and a retest) were generated by adding a randomly drawn measurement error to the “true” VA. The distribution of measurement errors was derived from ATS-HOTV protocol test-retest data reported in a previous study.⁷ The final simulated data consisted of both “true” and “observed” VAs of 10,000 subjects at baseline and each week of the 56-week follow-up for each of the four VA profiles (stable or improving VA over time for treatment with patching or spectacles). 

Rules to Determine the Status of VA

Six classification rules were defined to determine if a subject's VA stopped improving (Table 1). These rules were designed to reflect both the common clinical practice of using single measurements of VA over two or three visits and other rules that use the better or the average of test and retest VAs at each visit over two or three visits. Rules 1, 2, 3, and 4 used 0.1 logMAR as the threshold to define VA improvement, whereas rules 5 and 6 used 0.05 logMAR because these averaged the test and retest VAs. A detailed description of each rule is provided in Supplementary Appendix A2. 

Table 1.

 

Rules for Classifying Stability Based on the Observed VA

 

The follow-up schedule for the simulations was every eight weeks up to 56 weeks (i.e., one baseline visit and seven follow-up visits), which was designed to approximate some clinicians’ common clinical practice. The primary analysis was to determine the false-positive and false-negative rates for each of the six rules for detecting “stability” in amblyopic-eye VA during follow-up, as defined in Table 1. For the simulations of stable true VA, only the false-negative rate was calculated because the true VA status was always “stability” during the entire follow-up. 

For the simulations of improving VA, the rate of change in true VA at each follow-up visit was estimated by averaging the differences between the true VA at the current visit and the true VAs both at one and two weeks prior, and at one and two weeks after the current visit (except at one, 55, and 56 weeks, where only a subset of the aforementioned four VAs were available). If the estimated average rate of improvement was more than 0.005 logMAR per week (approximately 1/5 line per month), the true VA status was classified as still improving at that visit; otherwise, the true VA status was classified as stable at that visit. 

Starting from visit two (for rules based on two visits, i.e., Rules 1, 4, and 5) or visit three (for rules based on three visits, i.e., Rules 2, 3, and 6), each subject's observed VA status was determined to be “stable” or “still improving” by each classification rule based on the simulated observed VA. This determination was then compared with their true VA status. If a subject met the rule's criterion for stability while the true VA status was still improving, the individual was classified as a false positive for stability. If a subject met the rule's criterion for still improving while the true VA status was stability, the individual was classified as a false negative for stability (Supplementary Table A4). If a subject's VA status was identified as stability by the rule, no further follow-up visits were evaluated (because it was assumed that, in clinical practice, the subject would stop the current treatment and potentially start a new treatment); otherwise, evaluation of subsequent visits continued, and the subject's status was re-evaluated at the next visit. This process was repeated until either the VA met the rule's criterion for “stability” or the end of the simulated 56 weeks of follow-up was reached. 

The false-positive and false-negative rates for each rule were calculated by dividing the number of false-positive and false-negative cases by the total number of subjects in each simulation. In the case of a false negative (for both truly stable VA and truly improving VA), the time between the first false-negative occurrence and the first visit when the subject's VA was identified as stable by the rule was calculated to represent the delay of treatment. False-positive and false-negative rates for stability in VA and the average time of treatment delay resulting from false negatives (for stable or truly improving VA) were then compared between rules. 

Sensitivity Analyses

Several sensitivity analyses were performed to evaluate the effect of varying parameters and assumptions used in the simulations would have on the performance of the rules. First, the observed VA was simulated with a 50% increase and then with a 50% decrease in the original measurement error estimate. The magnitude of measurement error affected how far the observed VA could deviate from the true VA, and the overall performance of the rules was expected to improve with a decrease in measurement error. Second, two different follow-up schedules with either longer (every 12 weeks up to 56 weeks) or shorter (every four weeks up to 56 weeks) time between visits were used. Third, the effects of different rates of VA improvement per week (with a 20% increase and a 20% decrease) in determining the true VA status on the performance of the rules were also evaluated. Fourth, the simulations for patching and spectacles had differing mean, standard deviation, and range of starting VAs (derived from previous studies) and therefore provided a sensitivity analysis of starting VA. All simulations and analyses were conducted using SAS version 9.4 (SAS Inc., Cary, NC, USA). 

Results

Simulation of Visual Acuities

Visual inspection of the mean true VAs simulated for both the patching and spectacle treatment cohorts confirmed that they followed the profile of the actual measured VAs from the previous studies of patching treatment (Fig. 1A)^2–5 and of spectacle treatment⁶ (Fig. 1B). The observed VA, which incorporated randomly sampled measurement error, was analyzed with a precision of 0.1 logMAR, because the ATS-HOTV algorithm⁸ uses 0.1 logMAR increments. Examples of the simulations over the 56-week follow-up are illustrated for two of the 10,000 subjects from the patching treatment cohort (Fig. 2A) and for two of the 10,000 subjects from the spectacle treatment cohort (Fig. 2B). Simulations for both the observed and true VAs are shown. The change in color for the simulated true VAs indicates when the estimated rate of VA improvement was ≤0.005 logMAR per week as described in the methods. 

Figure 1.

 

Mean simulated true VAs and mean amblyopic-eye VAs from the previous PEDIG Studies. The mean simulated true VA is plotted over time along with the mean HOTV VAs of subjects enrolled in previous PEDIG studies treated with patching (A) and spectacles (B). The interquartile range is plotted above and below the line of mean simulated true VA. In addition, the mean simulated true amblyopic-eye VAs from subjects with stable VA over time are plotted for each treatment modality. (The 95% confidence interval of mean HOTV VAs of subjects enrolled in previous PEDIG studies are provided in Supplementary Table A5.)

Figure 2.

 

(A) Simulated true and observed VAs of two subjects with improving true VA with patching treatment. The simulated true VAs from two subjects with patching treatment are plotted for improving VA and when VA is stable. The observed VAs (with measurement error) of the two subjects are plotted along with the true VAs. The mean true amblyopic-eye VA simulated of all subjects with patching treatment is also shown (dotted line). (B) Simulated true and observed VAs of two subjects with improving true VA with spectacle treatment. The simulated true VAs from two subjects with spectacle treatment are plotted for improving VA and when VA is stable. The observed VAs (with measurement error) of the two subjects are plotted along with the true VAs. The mean true amblyopic-eye VA simulated of all subjects with spectacle treatment is also shown (dotted line).

Performance of Rules for Classifying Stable VA

The false-positive rates for “stability” for truly improving VA and false-negative rates for “stability” for truly stable VA for each of the six rules are provided in Table 2 and Figure 3. Also included are the false-negative rates for “stability” for truly improving VA (when the estimated rate of improvement was ≤0.005 logMAR per week). The average treatment delay for each rule is also provided in Table 2. The rules that required three visits to determine VA improvement status (2, 3, and 6) had very low false-positive rates but relatively high false-negative rates and longer treatment delays. In contrast, the rules that required only two visits to determine VA improvement status (1, 4, and 5) had higher false-positive rates but lower false-negative rates for stability and shorter treatment delays. 

Table 2.

 

Performance of the Rules Classifying Stability in VA^*

 

Figure 3.

 

Performance of the rules classifying stability in VA. The estimated false-positive and false-negative rates for the six rules estimated for improving true VA over time for patching treatment (A) and spectacle treatment (B). Also shown are the false-negative rates for stable VA over time for the six rules estimated for patching treatment (A) and spectacle treatment (B). For each rule, the percentage of each rate is shown within the bar and the height of the bar provides an indication of the rule's overall performance.

Sensitivity Analyses

The results of the sensitivity analyses (Supplementary Appendix A3) suggested that the overall performance of the rules improved as the measurement error decreased and vice versa (Supplementary Table A1; Supplementary Fig. A1). In the simulated data, the magnitude of improvement in true VA decreased over time, and therefore the follow-up schedule with more frequent visits (every four weeks vs. eight weeks) had a higher false-positive rate (for “stability”) because subjects were seen more frequently during the later stage of treatment when the true VA status was still improving but at a slower rate (Supplementary Table A2, Supplementary Fig. A2). Similarly, using a lower rate of improvement (0.004 logMAR/week vs. 0.005 logMAR/week) as the threshold for true VA improvement increased the false-positive rate (for “stability”) because more subjects were truly still improving later into the follow-up period (Supplementary Table A3; Supplementary Fig. A3). The false-positive and false-negative rates of the rules with the patching treatment profile and the spectacle treatment profile were similar, and therefore the starting VA did not significantly affect the performance of the rules. The relative performance of the six rules, assessed by the false-positive and false-negative rates, were not influenced by measurement error, visit schedule, threshold of “stability” for true VA status, or starting VA. 

Discussion

It is a common, long-standing practice for clinicians to compare a single VA measurement between two consecutive visits to decide whether VA is improving in patients being treated for amblyopia. This method of comparing single measurements of VA between two consecutive visits has also been used to determine when VA is no longer improving in randomized clinical trials.^1,9,10 “Stability” in VA is often defined as a difference of less than 0.1 logMAR between two consecutive visits. This clinical practice, tested as Rule 1 in the present study, has a reasonable balance between false-positive and false-negative rates (i.e., 38% and 30% for the simulations of patching, Table 2) in its ability to detect stability in VA when comparing the four rules using a single VA measure at each visit. A false-negative rate (declaring improvement when, in fact, there is stability) of 30% when tested against known nonimprovement for patching may be an acceptable rate for keeping a patient or research subject in treatment when VA is not improving. However, the relatively high (38%) false-positive rate (declaring stability when there is, in fact, improvement) for this simple rule, when tested against known improvement for patching, may be an unacceptable rate of classifying a patient with improving VA as stable in a research protocol because a study-specified intervention might be stopped or changed prematurely. Results were similar when improving and stable VA profiles with spectacles (as an alternative treatment) were simulated. 

As alternative approaches to Rule 1 (comparing single measures between consecutive visits), adding a third visit and comparing the VAs across the visits (Rules 2 and 3) markedly reduced the false-positive rate (declaring stability when, in fact, there is improvement) for conditions of known VA improvement; however, these rules increased the false-negative rate (declaring improvement when in fact there is stability) for conditions of known stable VA. The high false-negative rate for stability (43% for Rule 2 and 57% for Rule 3 for conditions of stable VA) would keep a patient or research subject in their current treatment when VA is stable and thereby delay a change in treatment. 

An alternative approach of adding a VA retest at each of the two visits and comparing the better VA from each visit (Rule 4) had very similar results to Rule 1. In contrast, conducting a test and retest at each visit and comparing the average VA from each visit (Rule 5) decreased the false-positive rate (declaring stability when there is in fact improvement) from 38% to 22% with known improvement from patching and from 35% to 21% with known improvement from spectacles; it only slightly increased the false-negative rate (declaring improvement when in fact there is stability) against known stability (from 30% to 35% and 29% to 36% for patching and spectacles, respectively). Although the false-negative rate increased when VA was truly improving with Rule 5, the delay in initiating or changing treatments was only slightly increased. 

Finally, comparing average VA across three visits (Rule 6) resulted in a false-positive rate of 0% for both patching and spectacles, but the false-negative rate for stability increased to 65% and 67% for patching and spectacle treatments, respectively, and resulted in the longest delays in initiating or changing treatments. Comparing the six rules, Rule 1 (single VA test, two visits) and Rule 5 (average of two VA tests, two visits) have the best balance between false-positive and false-negative rates and minimal delays in starting or changing treatments. Rule 1, requiring a single measure of VA at each visit and reflecting common clinical practice, minimizes delays in initiating or changing treatments but has a higher false-positive rate, which could lead to prematurely adding or stopping treatment. Rule 5 (using the average of test and retest) may be preferable for a research protocol in which a lower false-positive rate for stability is most desirable without introducing a significant delay in starting or changing treatment. 

These results of simulations performed in the current study depend on assumptions that include the sampled distribution of VA, the time course for VA improvement, and measurement error. A strength of this study is that the assumptions used for these simulations were based on systematically collected data from previous PEDIG amblyopia treatment studies^2–6 using HOTV isolated, crowded optotypes presented using a standard algorithm⁸ with a precision of 0.1 logMAR. The underlying pattern of true VA improvement was based on the profile of mean VA improvement in previous PEDIG studies,^2–6 and the measurement error was sampled from a distribution based on test-retest data from previous PEDIG studies.⁷ 

To evaluate the robustness of our findings, sensitivity analyses were conducted to determine how the magnitude of the measurement error used to simulate the observed VA, the time between visits, the rate of improvement in determining the true VA status, and the simulated starting VA impacted the false-positive and false-negative rates for the different rules. The relative performance across the rules was the same, regardless of how these parameters were varied. Nevertheless, changes in these assumptions affected the individual performance of each rule as would be expected. For example, decreasing the time between follow-up visits from eight weeks to four weeks increased the false-positive rate for stability when the true VA was improving over time. This suggests if a low false-positive rate for stability is desirable, specifying an eight-week interval between visits would be more advantageous than four-week follow-up intervals. Decreasing the magnitude of measurement error resulted in a decrease in both the false-positive rate for stability when true VA was improving and the false-negative rate for stability when VA was truly stable. This finding confirms the importance of continuing to investigate ways to reduce measurement error when assessing VA. Although beyond the scope of the current simulations, assessing VA with the e-ETDRS (in children aged seven and older) with 0.02 logMAR, single-letter increments would be expected to increase the precision of the VA measurement, reduce the measurement error (both because the test is more precise and older children and adults maybe provide more-accurate responses than younger children) and decrease the false-positive and false-negative rates as confirmed by the sensitivity analysis. 

This study has some limitations. These simulations are based on VA assessments in children 3 to less than 7 years of age with amblyopia measured by the ATS HOTV⁸ protocol, and may not be generalizable to other tests of VA. Simulations were also limited to improving or stable VA and may not apply to other conditions where VA could worsen over time. Based on our sensitivity analysis, however, the relative performances of the six rules explored here were robust to different ranges of VA, magnitude of measurement error, rates of improvement, and frequency of visits. The analyses were also limited to evaluating six rules for stability in VA. Future studies might explore additional rules that might yield lower false-positive or false-negative rates. However, the rules in this study were selected based on the feasibility of implementing them clinically or in a research protocol where precision and efficiency, like false-positive and -negative rates, are frequent tradeoffs. 

Results of the simulations presented here allow comparison of the performance of six different rules to make choices when considering changing an amblyopia treatment strategy (e.g., starting or stopping patching) by assessing the trade-offs between false-positive and false-negative rates in the context of the condition and consequences. If the greatest weight is placed on correctly classifying a research subject whose amblyopic-eye VA is improving (a low false-positive rate for stability) while still maintaining a reasonable false-negative rate for stability, Rule 5 (average of test and retest at a visit is <0.05 logMAR better compared with the previous visit average of test and retest) would be preferable. This rule might have been preferable to Rule 1 when used in a previous spectacle baseline run-in phase for a randomized trial of patching versus continued spectacles¹ because we found children randomized to continued spectacles, defined as not improving at randomization, had a subsequent mean improvement of 0.05 logMAR, indicating that they may have been continuing to improve. In contrast, there are other situations where it may be more important to determine when the treatment is not working, such as when treatment adherence is difficult, or the treatment imposes some risk. In such situations, it may be important to select a rule that has a low false-negative rate for stability to correctly identify true nonimprovement and minimize delays in stopping the treatment. In that case, Rule 1, the simple rule of comparing a single VA tested at each of two subsequent visits, would serve that purpose. 

In summary, the choice of a specific rule depends on the relative importance of correctly declaring stability versus an improvement in VA. This choice would also depend on factors specific to the clinical situation or study design, such as the risks of the treatment and the study question. 

Acknowledgments

Supported by the National Eye Institute of National Institutes of Health, Department of Health and Human Services EY011751, EY023198, and EY018810. The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. 

Disclosure: R. Wu, None; R.E. Manny, None; J.M. Holmes, None; B.M. Melia, None; Z. Li, None; D.K. Wallace, None; E.E. Birch, None; R.T. Kraker, None; S.A. Cotter, None