Abstract
Purpose: :
The vitreoretinal literature contains studies using the best visual acuity (VA) over several post-intervention visits as outcome variables. There is inherent test-retest variability in VA measurement; we hypothesized using repeated measures of VA and selecting the best one might bias the conclusion of a clinical investigation. We sought to quantify the direction and magnitude of this bias.
Methods: :
We tested this hypothesis using two different techniques. First, we performed Monte Carlo simulations on hypothetical studies of 10, 25, 50, and 500 patients. Each study size was subjected to 10,000,000 simulations. The type I error was defined as the frequency of rejecting the null hypothesis when there was no true difference in the outcome variable. Second, we constructed theoretical models using probability distribution functions (PDF) to prove_using fundamental laws from statistics and calculus_that the best visual acuity PDF becomes separated from the actual visual acuity PDF using clinically relevant parameters
Results: :
The Monte Carlo simulations demonstrated a consistent and significant bias toward erroneously concluding an improvement in vision when no true vision change occurred. This bias increases with increasing sample size and increasing number of visits from which the best VA is chosen. For studies with as few as 10 patients obtaining post-intervention VA from the best of 2 visits, the type I error was 24% (compared to 5% which is used in the majority of statistical testing). That means studies using this design with this size would support an incorrect conclusion 24% of the time. In studies with 50 patients obtaining post-intervention VA from the best of 2 visits, the type I error was 87%. In studies with 10 patients obtaining post-intervention VA from the best of 4 visits, the type I error was 65%. PDF models confirmed the magnitudes and directions of these errors, and provide the theoretical basis of these findings
Conclusions: :
Using the best VA selected from 2 or more post-intervention encounters significantly biases study results toward erroneously concluding an improvement in visual acuity post-intervention when none truly exists. Ineffective interventions would be mistakenly concluded as effective. These results could have an immediate impact on researchers, reviewers, editors, and readers.
Keywords: clinical (human) or epidemiologic studies: biostatistics/epidemiology methodology • clinical research methodology • visual acuity