Ranked Importance of Visual Function Outcome Measures in Choroideremia Clinical Trials

Abstract

Purpose: Clinical trials of novel therapies for choroideremia require robust and clinically meaningful visual function outcome measures. Best-corrected visual acuity (BCVA) is mostly insensitive to changes in disease state, until late stages, and hence also to potential therapeutic gains after gene therapies. While the insensitivity of BCVA as an effective outcome measure is common wisdom, its low importance has not been rigorously demonstrated in the literature. This work uses statistical techniques to rank the relative importance of potential functional outcome measures in choroideremia.

Methods: Retrospective dominance analysis was performed on data from a longitudinal interventional clinical trial performed at the Oxford Eye Hospital. Functional data from the untreated eye were analyzed and correlated with an anatomic measure of disease progression in the form of blue fundus autofluorescence area of the surviving outer retinal island. Each functional measure was then ranked in terms of variable importance.

Results: Microperimetry was the functional measure ranking first in terms of variable importance, followed by time since baseline visit, Pelli–Robson contrast sensitivity, high spatial frequency contrast sensitivity function, and low luminance visual acuity. Early Treatment Diabetic Screening chart BCVA under standard lighting conditions was ranked lowest among the panel of test modalities.

Conclusions: Clinical trials in choroideremia for early and mid-stage disease would be justified in moving away from using standard BCVA as a clinical trial outcome measure as we have shown its sensitivity to be relatively low compared to microperimetry. We have also demonstrated how functional measures rank in terms of importance.

Choroideremia is an X-linked retinal degeneration that leads to severe sight loss usually by middle age. A loss-of-function mutation in the CHM gene causes deficiency in Rab escort protein 1 (REP1), essential for the regulation of intracellular vesicular trafficking.¹ Functional measures form key primary or secondary endpoints in most clinical trials involving novel treatments for inherited retinal disease (IRD), including recently concluded gene therapy trials for choroideremia.^2,3 With some exceptions, the US Food and Drug Administration (FDA) has generally advocated the use of visual function outcomes over structural outcome measures such as retinal imaging obtained through optical coherence tomography (OCT) or blue light fundus autofluorescence (BAF)^4,5 for regulatory approvals. This is based on the correlation between significant visual acuity gain (generally considered 15 Early Treatment Diabetic Screening [ETDRS] letters) and improved quality of life. As we obtain more in-depth understanding of the pathophysiology underlying retinal disease and gain insight into the functional significance of various structural measures, such as BAF in geographic atrophy, the FDA has demonstrated increasing willingness to consider objective structural outcome measures. For instance, in glaucoma, intraocular pressure is accepted as a clinical trial endpoint after being validated as a direct correlate to visual field loss, as measured by static perimetry. In a more recent example, the reduced rate of expansion of geographic atrophy area as measured by BAF was the primary endpoint underpinning recent FDA approval of two complement factor inhibitors for dry age-related macular degeneration.⁶ However, most clinical trial primary endpoints comprise functional outcome measures. (For the pivotal clinical trial for Luxturna for RPE65-associated leber congenital amaurosis (LCA), the multiluminance mobility test was prevalidated and used as a primary endpoint, which enabled FDA approval of the first retinal gene therapy.⁷) As a result, many IRD gene therapy programs are exploring low luminance-based functional tests as potential endpoints. In this study, we explore a range of functional vision testing modalities. 

Historically, best-corrected visual acuity (BCVA) has been considered the gold standard clinical trial functional outcome measure. Indeed, key early clinical trials involving anti-VEGF for the treatment of neovascular age-related macular degeneration (AMD)^8,9 and trials for the treatment of diabetic retinopathy^10,11 used visual acuity as the primary endpoint and led to approval due to measured improvements in acuity. However, several IRD clinical trials utilizing BCVA have subsequently failed to meet their primary endpoints¹² (see reference 13 Table 1 for choroideremia trials failing to meet their visual acuity primary endpoint¹³). It is now well understood that for many rod-cone dystrophies, the disease presents at the region of highest rod densities: the mid-periphery. In choroideremia in particular, patients typically present with nyctalopia and constriction of their visual field,^2,14 but visual acuity is often minimally affected in contrast to the situation in AMD. Thus, researchers have turned to alternative functional tests that may better detect disease progression and potential treatment-induced gains. A few studies^13,15,16 provide an extensive overview of clinical trials and outcome measures in choroideremia and a range of other inherited retinal dystrophies. 

Table 1.

 

Baseline Statistics for All 12 Patients With Choroideremia Included in the Analysis

 

Table 2.

 

Number of Visits Included for the Dominance Analysis for Each of the 12 Patients With Choroideremia

 

During clinical trial design, it is imperative that the most appropriate outcome measures are identified. Selection of sensitive outcome measures would reduce patient burden (in terms of the number of unnecessary tests being performed), increase trial success rate in the presence of treatment efficacy, and reduce overall costs. It would also mitigate against trial failures associated with inappropriate selection of outcome measures. To this end, phase 1 and 2 clinical trials for IRDs, which primarily test for safety, often include a suite of functional tests performed as secondary outcome measures. These serve as exploratory endpoints informing what functional measure may be best for future phase 2/3 clinical trials. This work analyzes data from a recent phase 2 choroideremia clinical trial (Regenerate Study, ClinicalTrials.net ref: NCT02407678) and uses a statistical technique known as dominance analysis^17,18 to directly compare, in a robust manner, a range of functional measures. Dominance analysis is an extension of multiple linear regression or linear mixed-effect modeling where a brute-force evaluation of every combination of included predictors is performed and each R² fit is compared to deduce each predictor’s contribution to the overall model fit. Dominance analysis results in a robust evaluation of variable importance, even in the presence of highly correlated predictors (multicollinearity). 

The aim of this study is to provide evidence basis to help identify the most relevant functional measures that correlate well with disease progression and reduce burden on clinicians and patients by limiting the number of unnecessary test modalities. Identification and validation of the most appropriate visual function endpoint for choroideremia would improve the success rate of future therapeutic trials for this blinding condition. 

Methods

Retrospective analysis of the nontreated eye of 12 patients with choroideremia enrolled in a phase 2 clinical trial (Regenerate—NCT02407678) was conducted.³ Data collected as part of the trial included were microperimetry (MP); mean sensitivity using the MAIA microperimeter (ICARE; Mainline Instruments Limited, Birmingham, UK) with the standard 68-point 10-2 grid and 4-2 strategy; BCVA using the ETDRS chart; low luminance visual acuity (LLVA) using the ETDRS chart; Pelli–Robson contrast sensitivity; spatial contrast sensitivity function (CSF) at spatial frequencies of 1, 4, 8, and 10 cycles per degree using the Metropsis visual function assessment software¹⁹; the low vision Cambridge color test²⁰; and short-wavelength BAF captured using Heidelberg Spectralis OCT (Heidelberg Engineering GmbH, Heidelberg, Germany). BAF area (referring to the area of residual hyper-autofluorescent retinal islands) was obtained using the inbuilt caliper tool within the Heidelberg Heyex viewing software. Area measurements, by default within Heyex, use normative values for axial length and optical magnification when converting the angular measurements into mm² area measures. More precise area measures would require knowledge of axial length and optical magnification at the individual participant level. However, since we are primarily interested in longitudinal changes in BAF within each patient, absolute area measures are not required, and area changes from baseline provide an accurate and consistent assessment of BAF changes so long as each BAF image is captured using the follow-up function within Heyex. Where several disconnected BAF islands were present, only the area of the island encompassing the central fovea (and any contiguous regions) was considered. 

For statistical analysis, BAF was set as the dependent variable and considered a valid surrogate measure of disease severity.^21,22 All other variables were set as predictor/independent variables. R² fit is a measure of the degree of variance of data about the model fit in a linear regression. Analysis of variable importance was performed using dominance analysis (Shapley regression) with linear regression conditional R² fit used for determining variable importance within a linear mixed-effect model framework, where patient was set as the random intercept variable, to correctly model the incomplete longitudinal repeated-measures data set. The conditional aspect summarizes the model fit, taking within-subject variance, as well as between-subject variance, into account. All statistical analysis was performed in R (v4.2.1)²³ and utilized the Domir package²⁴ to perform the dominance analysis and the lme4²⁵ package to perform the linear mixed-effect modeling. Confidence intervals were obtained using a brute-force bootstrap method to generate 2500 samples from which the 95% confidence intervals were extracted. Model description and checks are provided as supplementary material, and the full code used for model selection and dominance analysis is provided on GitHub (https://github.com/amanasj/Dominance_analysis). 

Results

The baseline mean (SD) characteristics are outlined in Tables 1 and 2. Microperimetry tests all met the reliability criterion of <15% fixational losses, and all patients exhibited good foveal fixation with a mean (SD) 95% bivariate contour ellipse area of 3.04 (3.78) mm². 

Where available, data from all 12 visits for each patient (baseline and months 1, 3, 6, 9, 12, 18, 24, 30, 36, 48, 60) were included in the dominance analysis. However, during the COVID-19 pandemic, four of the earliest recruited patients missed up to three scheduled visits between months 30 and 48. In many of these cases, the visits were performed within an acceptable interval outside of the observation window, but in one case, a patient withdrew early (month 30). As per the trial protocol, BCVA and microperimetry and BAF were collected at every visit for the entire 5-year trial, and LLVA was collected at every visit between baseline and year 2 but was not extended to the full 5-year follow-up trial. Pelli–Robson contrast sensitivity, spatial contrast sensitivity function, and low vision Cambridge color test were collected at baseline and months 6, 12, and 24. 

The overall fit of the linear mixed-effect model, which included all the predictor variables, had an extremely high conditional fit statistic of R² = 0.996, as would perhaps be expected when considering so many explanatory variables. The strongest associations between disease severity (as inferred by size of hyper-autofluorescence area on BAF) and all possible covariates were with time (since baseline visit) (R²= 0.15; 95% confidence interval [CI], 0.09–0.27) and microperimetry (R² = 0.13; 95% CI, 0.02–0.23). This suggests that time is significantly correlated with changes in BAF with 15% of variance in BAF area measurements being explained by time progression alone. Similarly, microperimetry is also significantly correlated with changes in BAF explaining 15% of variation in BAF area measurement. As a result, microperimetry was ranked second as the predictor of most importance. Figures 1 and 2 show example patient results for a selection of longitudinal BAF and microperimetry images. The microperimetry reduces as the area of BAF reduces with disease progression over the 5 years of monitoring, but BCVA remains fairly static, only demonstrating an initial decline once the disease encroaches the fovea (as can be seen at month 60). 

Figure 1.

 

Selection of longitudinal BAF and microperimetry images and BCVA for an example patient included in the dominance analysis. In this instance, microperimetry mean sensitivity can be seen to decline from baseline to month 60, along with the hyperfluorescent area outlined. BCVA was relatively maintained throughout and at month 60 only dropped by seven letters from baseline, marking the onset of foveal involvement. LLVA and Pelli–Robson tests were not part of the protocol and so not performed after month 24 and, in this case, did not demonstrate a decline in line with BAF.

Figure 2.

 

Data from a second example patient. Here we can see the preserved BAF from baseline through to month 6 and corresponding relative preservation of MP mean sensitivity, BCVA, LLVA, and Pelli–Robson contrast sensitivity. By contrast, in the following 6-month period (M6 to M12), the disease appreciably progresses, as defined by BAF. During this same period, only MP mean sensitivity and LLVA demonstrate similar reductions.

Figure 3.

 

Bar chart with the relative contributions of each visual function outcome measure toward the overall conditional R² linear mixed effect model fit and 95% confidence interval error bars. The greater the degree of variance (in BAF) explained by each predictor, the higher the relative variable importance ranking position.

Pelli–Robson contrast sensitivity test was ranked third, and high spatial frequency CSF was ranked closely behind at fourth, with spatial frequency gratings of 10 cycles/degree, in particular, warranting further investigation. Notably, BCVA was ranked 12th, being last in terms of predictor importance and explaining only 0.9% of the variance in BAF. 

Discussion

This study supports the use of microperimetry in studies of choroideremia and demonstrates the limitations of BCVA as a functional endpoint due to its relative insensitivity compared with microperimetry in early- to mid-disease stage patients, with BCVA here being ranked the lowest variable of importance. In later disease stages, when foveal involvement may be present, BCVA may be a more appropriate and sensitive functional measure, as previously shown in a phase 1/2 interventional trial.² Time (since baseline visit) as a predictor was strongly associated with disease progression as would be expected, with microperimetry ranked second as a predictor of BAF area. Of note is the interesting finding of relatively high rankings of various contrast sensitivity tests. In particular, the Pelli–Robson test and high spatial frequency contrast sensitivity function may be promising future outcome measures as they demonstrate greater correlation with anatomic progression in the mid-disease stage than BCVA. The Pelli–Robson and LLVA tests are nondemanding and highly repeatable functional tests, with many of the attractive attributes of standard ETDRS visual acuity testing, and therefore may be easily implemented as outcome measures across multiple study sites. While the lower spatial frequency contrast sensitivity function and color vision tests (protan and tritan tests) may have their utility, they were relatively low on the ranking list and, for patients with inherited retinal disease enrolled in interventional clinical trials, may not demonstrate a strong enough association to warrant the time taken to perform them. However, we stress that relative rankings of Pelli–Robson, LLVA, spatial contrast sensitivity function, and Cambridge color tests are very similar with overlapping confidence intervals, all individually explaining between 5% and 10% of the variation in BAF. In addition, since these tests were performed only intermittently, these relative rankings may be prone to dramatic change if more data were to be collected and over a longer duration. However, BCVA, microperimetry, and BAF were collected at every visit throughout the full 5 years, so the comparative rankings of BCVA and MP can be viewed as accurate. Indeed, when taking into account the bootstrapped 95% confidence intervals, the only pairwise contrasts that reach the threshold of significance are between BCVA and all other tests, implying the 12th-place ranking of BCVA is statistically significant. While microperimetry is ranked second, its wide 95% confidence intervals imply that further data are required to establish this ranking to statistical significance. 

The ranking of BCVA in last place in this cohort of patients (attributing to only 0.9% of the variation in BAF area) is a clinically significant finding and in keeping with the authors’ previous research.^26,27 This definitively demonstrates the utility of microperimetry over that of BCVA to detect changes in disease state in this cohort as measured by BAF and, therefore, microperimetry's ability to detect treatment efficacy in clinical trials. Indeed, since BCVA is only affected when the edge of RPE and outer retinal degeneration reaches the fovea in late-stage disease, it would only be expected to be a useful clinical trial endpoint in patients with end-stage choroideremia, in whom there exists some degree of foveal involvement. However, these patients may then be too far progressed in terms of RPE and photoreceptor dysfunction in the central macula region for gene augmentation to halt further disease progression due to loss of trophic support and retinal inflammation. These factors would be in addition to the surgical challenges and dosage variability related to subretinal vector injections in patients with small central BAF islands.¹⁸ 

Here, we have demonstrated dominance analysis as a methodology for determining how various functional measures rank in terms of importance using a recent interventional clinical trial in choroideremia as an example. In this example, we find that standard visual acuity likely has limited utility as an outcome measure in early- to moderate-stage choroideremia, and our analysis favors microperimetry over visual acuity to maximize the possibility of treatment detection as implied by changes in BAF. Future trials in choroideremia may find that contrast sensitivity function is also a valuable outcome measure, and our dominance analysis suggests further research in this area is warranted. Furthermore, further work to incorporate structural measures into a dominance analysis is under way to investigate if changes within the smooth zone or outer retinal tubulations within the BAF area improve associations with functional measures. The smooth zone has previously been shown to correlate well with microperimetry and so may provide an improved biomarker for disease progression,²⁸ and it may prove to be a more sensitive structural measure in early-stage disease. The primary limitation of this study, and possibly its wider adoption in future studies, is the large sample sizes (both cross-sectional and longitudinal) required to generate a greater confidence in precise rankings. A second limitation is the possible presence of endogeneity with the included predictor variables. Correlation of error terms in linear regression with predictors could lead to erroneous conclusions, but it is not evident that endogeneity is a warranted concern here. Indeed, while assumptions such as normality and homoscedasticity are frequently examined before interpretation of results within eye research, endogeneity is rarely discussed. Further work is needed to investigate the potential for endogeneity within functional measures in retinal diseases. 

In conclusion, dominance analysis may be extremely useful to perform as a preliminary analysis in the design stage of clinical trials for retinal diseases to determine the optimal outcome measure. It is important to note that while the overall R² model fit statistic is usually of interest to researchers to determine whether the overall model is appropriate, the inclusion of a greater number of variables inevitably leads to inflation of R² due to potential multicollinearity and overfitting. Hence, dominance analysis is not intended for model selection but rather variable importance, once a model has been selected, evaluating each predictor’s contribution to the overall fit statistic. A model selection process, followed by dominance analysis, reveals the relevant functional predictor variables that should be assessed and determines their relative ranking of importance, respectively. This is analogous to power analyses in determining the optimal sample size during clinical trial design, which may be of great benefit as an additional tool for clinicians and researchers. Each retinal disease would likely yield different rankings depending on the exact pathophysiology, presentation, and progression, but using prior knowledge could potentially reduce the incidence of clinical trial failure due to the inappropriate selection of an outcome measure. 

Acknowledgments

Supported by the Oxford NIHR Biomedical Research Centre. 

Disclosure: A.S. Josan, None; L.J. Taylor, None; K. Xue, None; J. Cehajic-Kapetanovic, None; R.E. MacLaren, None