Abstract
Purpose :
When designing randomized clinical trials (RCT) for strabismus and/or amblyopia, stereoacuity is often an outcome measure. To allow early release, it is appealing to design an RCT with early failure criteria that simultaneously ends participation and assigns final outcome. We hypothesized that early failure criteria would yield a higher proportion of RCTs concluding false positive (FP) differences between treatments.
Methods :
We created hypothetical RCTs (treatment A vs B), 200 patients in each group, and an equal proportion of starting stereoacuity values (40, 60, 100, 200, 400 arc seconds, converted to log10 for analysis), over a 7-visit study. Treatment A had a linear improvement and B had increasing rate of improvement, with the same starting and ending values. We incorporated stereoacuity measurement error (noise), extracted from open source PEDIG IXT1 data (354 test-retest pairs). We simulated stereoacuity values at each visit based on starting stereoacuity and profile (A or B), adding randomly sampled noise to each value, rounding to the nearest measurable increment. We then repeated for 10000 RCTs and evaluated each RCT, with and without an early failure rule; worsening of 2 or more levels from baseline (assigning value at failure as final outcome). For each RCT, we calculated mean difference and 95% CI (A vs B), with and without the failure rule, determining proportion of RCTs where 95% CI did not include zero (false positive RCT). We represented precision of proportions using 95% CIs from binominal distributions and compared by Fisher exact.
Results :
A greater proportion of 10000 simulated RCTs had false positives with the early failure rule than without (5.49%, 95% CI 5.05% to 5.94% versus 0, 0%, 95%CI 0% to 0.000001%, difference 5.49% p<0.0001). In sensitivity analyses, “without early failure” not approaching 5% FPs was due to the bounded and narrow distribution of our derived noise, and non-normal underlying data. Unbounding and widening distributions of noise increased the proportion of FPs; proportions were always greater with the early failure rule than without.
Conclusions :
Using an early failure rule for stereoacuity outcomes increases risk of erroneous RCT conclusions. This risk occurs when there is both variability and rate of change differs between treatments. Designers and readers of clinical trials need to be aware of this risk.
This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.