The Use of Best Visual Acuity over Several Encounters as an Outcome Variable: An Analysis of Systematic Bias

Abstract

Purpose.: To investigate whether the use of the best of multiple measures of visual acuity as an endpoint introduces bias into study results.

Methods.: Mathematical models and Monte Carlo simulations were used. A model was designed in which a hypothetical intervention did not influence the visual acuity. The best of one or more postintervention measures was used as the outcome variable and was compared to the baseline measure. Random test–retest variability was included in the model.

Results.: When the better of two postintervention measures was used as the outcome variable with a sample size of 25, the model falsely rejected the null hypothesis 55% of the time. When the best of three measures was used, the false-positive rate increased to 90%. The probability of falsely rejecting the null hypothesis increased with increasing sample size, also increasing the number of measures used to select the outcome variable.

Conclusions.: Using the best of multiple measures as an outcome variable introduces a systematic bias resulting in false conclusions of improvement in that variable. The use of best of multiple measures of visual acuity as an outcome variable should be avoided.

Visual acuity is a common outcome variable in clinical ophthalmology research. DiLoreto et al. ¹ reviewed 1 year's worth of publications from three major American ophthalmology journals looking for articles in which best and/or final visual acuity was used as an outcome measure. “Best” vision refers to the practice of selecting the maximum vision among multiple measurements. Among articles reporting visual acuity as an outcome measure, best vision was used in 3.6%. Most of those were small studies. ¹ Best vision is most commonly used as an outcome in retrospective studies exploring the effect of an intervention. Vision measured before the intervention is used as the baseline variable, and the best of several postintervention measurements is used as the outcome variable. DiLoreto et al. ¹ discouraged the use of best vision as an outcome, because of the possibility of introducing bias and indicated that this bias could lead to underestimation or overestimation of the visual outcome, depending on the trend in visual acuity change over time. 

Studies have consistently shown that even under ideal testing conditions, there is test–retest variability in the measurement of visual acuity. ^2–5 We hypothesized that this variability would introduce bias that could cause erroneous conclusions when using best vision as an outcome measure. We explored the nature of this bias, using both theoretical calculations and numerical simulations. 

Methods

We first derived theoretical calculations of the probability distribution function produced by selecting the largest value of multiple random measurements. We assumed that the measurements had a normal distribution. Repeated visual acuity measurements, at least when performed with a logMAR chart, approximate a normal distribution. ^2,4,5 These calculations were then verified by using a Monte Carlo technique to simulate repetition of this experiment millions of times and to compare the theoretically calculated with the simulated results. 

We also created a hypothetical experimental model in which the visual acuity was unchanged by an intervention. A single, baseline acuity was measured and compared to the best of one or more acuity measures after the intervention. 

Part I: Calculation of Probability Distribution Functions

We first present the theoretical calculations of the probability distribution functions. Let the random variables v ₁, … , v_n represent n repeated measurements of visual acuity. Assume each measurement v_i is a random selection from a normal distribution (representing the normal variability in repeat testing of visual acuity). For simplicity, assume the mean and SD are 0 and 1, respectively. The probability distribution function (pdf) for v_i , defined as p_i , is defined as:  

 

 

Then the probability that any given v_i is less than or equal to some value x, denoted P_i (x), is defined as the cumulative distribution function. Thus P_i (x) is given by  

 

 

Now consider V _max, the maximum of the n acuity measurements. The probability that V _max ≤ x, denotes P _max(x), is the probability that v_i is less than x for every v_i . If the individual v_i values are distributed independently, then it is the product of the probabilities that each individual v_i is less than x. Because they are distributed identically, it is equivalent to this probability for any v_i raised to the nth power. That is,  

 

 

Now P_max is the cumulative distribution function for V_max, the maximum of n acuity measurements. We are interested in the probability distribution function for V_max, denoted p_max. This function is simply the derivative of P_max with respect to x. That is,  

 

By the fundamental theorem of calculus, we have:  

 

P_max(x) was numerically integrated to calculate its mean and SD. 

To verify these theoretical results, we performed Monte Carlo simulations of the experiment just described. In a Monte Carlo simulation, a computer uses a random-number generator to simulate running a large number of independent iterations of an experiment. The results are then tallied to measure the frequency of various outcomes. To run these simulations, we used a commercial software package (MatLab; MathWorks, Natick, MA). Vision was considered a continuous variable, and the distribution of test–retest error was assumed to be normally distributed with mean and SD of 0 and 1, respectively. For each iteration, a random value was drawn from the test–retest distribution and assigned as the baseline vision. We then simulated taking the best of 2, 3, or 4 subsequent independent vision measures, using the same normal distribution for test–retest variation. For each experiment, we ran 10 million iterations; from these results, we could form a numerical estimate of p _max(x), and calculate its mean and SD. This estimate was then compared to the theoretical calculation of p _max(x). 

Part II: Simulated Experiments and Statistical Power

To evaluate the consequences of these probability distributions on experimental results, we analyzed hypothetical experimental designs. We created an experimental model in which the visual acuity was unchanged by a hypothetical intervention. A single, baseline acuity was randomly selected from the normal distribution. The outcome variable was the greatest of one or more random selections from the normal distribution. We then calculated the likelihood that statistical testing would detect a significant difference between the baseline and outcome visual acuities for various sample sizes. 

As in part I, we first used a theoretical approach by calculating the power of a paired t-test using our experimental design. In general, a statistical test's power is the probability that a statistically significant difference will be detected if one truly exists. For the preintervention vision, we assumed that the acuity was normally distributed with a mean of 0 and an SD of 1. For the postintervention vision, we used the probability distribution functions for the best of multiple random samples, as derived in part I. We set the level of statistical significance (type I error) at 0.05, as is common. The statistical powers were calculated by using the freely distributed software package G*power 3, as described by Faul et al. ⁶  

We then ran Monte Carlo simulations to numerically evaluate the outcomes. We created models of the experiments. Monte Carlo simulations were run with a minimum of 10 million iterations for each combination of sample size and outcome sampling rate. Paired t-tests with a critical P ≤ 0.05 were used to compare the baseline and outcome visual acuities. Summary statistics from the simulations were calculated (MatLab; MathWorks Inc., Natick, MA; or Excel; Microsoft Inc., Redmond, WA). 

Results

Table 1 shows the baseline theoretical standard distribution with mean of 0 and a SD of 1, and plots of the probability distribution functions obtained from the maximum of 1, 2, 3, or 4 samples drawn from the same distribution. The case for 1 sample is a control trial and shows that the Monte Carlo simulation gives the expected result. The distribution plots diverge further from the original population mean as one takes the best of a larger number of measurements. The plots from the theoretical and Monte Carlo methods were identical. The mean and SD as calculated from the theoretical p _max(x) and the Monte Carlo simulations are also presented. They are in close agreement. 

Table 1.

 

Probability Distributions Resulting from Choosing the Maximum of Multiple Measures of Visual Acuity

 

 The baseline visual acuity is assumed to be normally distributed, with a mean of 0 and SD of 1. The results of selecting the best of 1 (to validate the methods), 2, 3, and 4 independent measures of visual acuity are presented. The results from both the theoretical calculations of the probability distribution function and the Monte Carlo simulations are presented.

Table 2 shows the likelihood that a paired t-test would detect a difference between pre- and postintervention populations in the hypothetical experiment described earlier. This likelihood of rejecting the null hypothesis is presented for different combinations of sample size and variations of the best acuity measurement. We present the results from the theoretical study power calculations, as well as the results of the Monte Carlo simulations. As expected, if the outcome variable is based on a single measurement, the likelihood of rejecting the null hypothesis approximates 5% when the usual definition of statistical significance is used (critical P ≤ 0.05). When the outcome variable was the best of two repeated measures and with a sample population of 10, a paired t-test concluded that there was a difference 24% of the time. As sampling rate and sample size increase, the likelihood of finding a false-positive outcome increases sharply. 

Table 2.

 

Probability of Rejecting the Null Hypothesis for Various Study Conditions

	Number of Measures from Which Maximum Value Was Selected
	1			2			3			4
	Sample Size (n)			Sample Size (n)			Sample Size (n)			Sample Size (n)
	10	25	50	10	25	50	10	25	50	10	25	50
Theoretical study power, α	0.05	0.05	0.05	0.23	0.55	0.85	0.48	0.90	1.00	0.66	0.98	1.00
Proportion of simulations rejecting the null hypothesis	0.05	0.06	0.05	0.24	0.56	0.87	0.48	0.91	1.00	0.65	0.98	1.00

 Theoretical study powers were calculated by using a normal distribution as the baseline variable and the probability distribution functions from Table 1 as the outcome variables. For the Monte Carlo simulations, the baseline value for each subject was a random selection from a normal distribution. The outcome variable was the maximum of 1, 2, 3, or 4 random selections from the same normal distribution. A paired, two-sided t-test with critical P < 0.05 was used to determine statistical significance.

Discussion

It is not uncommon in the literature to find studies in which a single, preintervention, visual acuity is compared to the best of several postintervention visual acuities. ¹ However, as several studies have shown, there is variability in the repeated measurement of visual acuity, even under ideal testing conditions. ^2–5 We investigated the statistical bias this practice would induce, using both theoretical and Monte Carlo models. We achieved the same results with both techniques, helping to validate our conclusions. Our models reflect a situation in which baseline vision is based on a single measure, and outcome vision is based on the maximum of two or more measures from the same distribution. The systematic selection of the best of multiple measures introduces the false impression of improvement from baseline when none truly exists. This is driven by the asymmetric selection of the “best” result among multiple measure of a variable subject to test–retest variability. 

In our model, the probability of rejecting the null hypothesis and erroneously concluding a treatment benefit became high, even with modest sample sizes and a low number of postintervention measures. Similarly, in a real experiment, true treatment benefits could be exaggerated. In fact, analysis of an intervention with a harmful effect on acuity could be sufficiently biased to conclude a beneficial effect. The assumptions of our model may differ from those of any specific clinical situation. However, if there is any variability in repeatedly testing vision, using the best of multiple tests will always bias toward improved vision. The likelihood of finding a false-positive outcome increases as one increases the number of subjects and the number of postintervention measures. 

We have ignored patient-to-patient variability in our model. We have done this for simplicity, since this does not influence the conclusions of our simulations. We used a paired t-test in our model. The null hypothesis is that the average change in vision for the sample is 0. This analysis first calculates the difference between the outcome and baseline vision for each subject. Since the difference in acuity is calculated for each subject, baseline intersubject differences do not influence the analysis. Under these assumptions, the results of our models would be the same whether individual study subjects entered with similar or different visual acuities. 

These results are not specific to visual acuity, but would apply to any measure for which there is random test–retest variability. We chose a normal distribution for ease of visualization and because logMAR visual acuities do appear to have a normal test–retest distribution. However, the derivation of p _max(x) did not require a normal distribution, and, a similar equation could be derived for any initial probability distribution. One might observe that a paired t-test requires both test populations to have normal distributions. The distribution of a best acuity measure is not normal. We do not feel this affects our overall conclusions, however. Although the probability distribution functions for the best of multiple measures are not truly normal, they probably approximate normality more closely than most clinical data sets. The t-test is fairly robust to non-normality. Moreover, the use of a t-test reflects typical practice in the smaller studies that are more likely to use a best visual acuity measure. 

This model is illustrative of the bias introduced by using the best among two or more measures of visual acuity as an outcome in clinical research; however, there are limitations. The model does not account for many factors likely to be present in real clinical data. Among these factors would be drift in vision over time due to natural history and “floor” and “ceiling” limits at the extremes of visual acuity. Thus, the magnitude (but not the direction) of the bias will vary with real-world data. 

Using the maximum of multiple measures of visual acuity as an outcome variable introduces unacceptable bias favoring the false conclusion of improvement in the variable. Authors, journal editors, and peer reviewers should be aware of this bias so that the methodology can be avoided. 

Footnotes

 Supported in part by an unrestricted grant from Research to Prevent Blindness, Inc., New York, New York.

Footnotes

 Disclosure: D. Koozekanani, None; D.J. Covert, None; D.V. Weinberg, None