A New Epidemiological Aid in Deciding Whether to Continue or Stop a Treatment

Abstract

Purpose.: To present a new epidemiological method relying on randomized controlled clinical trial (RCT) data to assess whether a treatment was effective, aiding in the decision to continue or stop the treatment in clinical patients.

Methods.: A cutoff point is calculated in the change of a continuous outcome for which a proportion of treated patients clearly achieved a change better than this cutoff point as a result of the treatment. This cutoff point can then be applied to individual patients during routine therapy. The method was applied to reports of the Minimally Classic/Occult Trial of the Anti-VEGF Antibody Ranibizumab in the Treatment of Neovascular AMD (MARINA) trial, which included patients with AMD treated with monthly intravitreal injections of ranibizumab, and to reports of trials involving patients with high IOP, macular edema, and convergence insufficiency.

Results.: The cutoff point in the change in visual acuity (number of letters), above which a proportion of patients clearly benefited due to ranibizumab treatment, was −5.0 at 24 months follow-up. The proportion of treated patients who ended above this cutoff point due to the treatment was 60%. The cutoff point varies with time of follow-up and by subgroup.

Conclusions.: Contrary to common interpretation, no change, or a limited decline, in the outcome (visual acuity) can still imply that the patients are better off with the treatment than with no treatment. Stopping the treatment above the cutoff point may not be appropriate since it was effective in at least a proportion of patients. This method applies to a broad range of scales and conditions. (ClinicalTrials.gov number, NCT00056836.)

Introduction

In clinical practice, the decision to continue, discontinue, or change a treatment is a daily problem. In many cases this decision depends strongly on the interpretation of the change from baseline in an outcome such as visual acuity or IOP. Rational decision making implies that when the change in outcome indicates that the treatment has been effective, continuing the treatment is a logical decision. Similarly, discontinuing the treatment is appropriate when the treatment has not been effective. 

However, it is often difficult to decide whether a treatment has been effective, since the outcome of the natural course of the disease, had the treatment not been given, is not known for treated patients. It is attractive to follow one's intuition and to assume that no change in the outcome, or worsening of the outcome relative to baseline, is proof of no effect. However, this assumption is not supported by evidence. 

In this article we present a method to calculate the cutoff point for the change in a continuous outcome variable above which the treatment was clearly effective in a proportion of patients. As an example, we use data from a randomized controlled clinical trial (RCT) of intravitreal ranibizumab versus sham intravitreal injections in the exudative form of AMD. ¹ AMD is the leading cause of blindness in elderly people in developed countries. ² The disease plays a major role in the daily practice of many ophthalmologists. As an important measure of change in disease intensity, the level of change in visual acuity is the continuous outcome variable in AMD. In addition to this AMD example, we apply the method to other scales and conditions: IOP reduction with topical medication in elevated IOP, visual acuity in the treatment of refractory diabetic macular edema, and near point of convergence (NPC) in the treatment of convergence insufficiency. 

Methods

We defined normal distributions using results of randomized trials to calculate the cutoff point above which it is certain that a proportion of treated patients ended due to the treatment. Normal distributions commonly apply to values of observations that cluster around a mean. 

For four examples of ophthalmologic treatment, we derived from trial reports the mean change in outcome in the treated group and the nontreated reference group (μ_t and μ_r , respectively) and the SD of this change (σ_t and σ_r , respectively) (see Tables 1, 2, 3). We converted SDs from SEMs or 95% confidence intervals in the trial reports using the guidelines in the Cochrane Handbook for Systematic Reviews of Interventions. ³  

Table 1. 

 

Results for Five Times of Follow-up in the MARINA Trial

Follow-up (Months)	Change in ETDRS Visual Acuity, Mean (SD)*		Cutoff Point x†	At above Cutoff Point (%)‡	TAE above Cutoff Point (%)§
	Ranibizumab Group (n = 238)	Sham Group (n = 238)
1	3.9 (10.2)	−0.2 (8.6)	4.9	46	40
3	5.9 (10.5)	−3.7 (11.3)	0.4	70	49
6	6.5 (11.8)	−6.6 (13.0)	−0.9	73	55
12	7.2 (14.6)	−10.4 (15.1)	−1.9	73	61
24	6.6 (17.2)	−14.9 (18.8)	−5.0	75	60

 ETDRS, early treatment diabetic retinopathy study; SD, standard deviation.

*  Standard deviation calculated using standard errors (SE) from the trial report: SD = SE · n .

†  Point above which for every change in visual acuity a proportion of the treated patients achieve their change due to the treatment.

‡  Treated patients who ended above cutoff point.

§  Treated patients who ended above cutoff point due to treatment.

Table 2. 

 

Results per Effect Modifier in the MARINA Trial

Effect Modifier	Subgroup	No. in Treated/ Reference Group	Change in Visual Acuity at 24 Months, Mean (SD)*		Cutoff Point x†	TAE (%)
			Ranibizumab Group	Sham Group
Age, y	50–64	16/11	6.1 (21.2)	−13.7 (23.9)	−6.2	48
	65–74	64/67	7.2 (15.8)	−11.9 (19.7)	−4.8	54
	75–84	124/132	7.6 (16.4)	−16.0 (19.0)	−5.3	64
	≥85	36/28	1.9 (16.4)	−16.8 (19.3)	−9.4	54
Initial visual acuity	20/160 or worse	48/51	10.6 (17.5)	−0.8 (13.3)	9.1	57
	20/100 to 20/125	59/50	9.3 (15.4)	−13.6 (16.1)	−2.4	69
	20/63 to 20/80	68/72	5.4 (16.2)	−20.0 (17.6)	−7.7	69
	20/50 or better	65/65	1.8 (15.8)	−21.3 (19.8)	−11.4	61
CNV lesion size, DA	≤2	39/46	10.2 (14.2)	−13.4 (18.2)	−2.9	66
	>2 to ≤4	86/77	9.7 (14.4)	−15.5 (18.7)	−4.0	68
	>4 to ≤6	63/60	3.8 (20.0)	−15.0 (18.3)	−4.3	57
	>6	52/55	2.1 (16.7)	−15.5 (20.7)	−9.8	49
CNV lesion type	Minimally classic	91/87	6.4 (20.0)	−14.7 (17.3)	−2.6	64
	Occult	149/150	6.2 (14.7)	−15.3 (19.5)	−6.6	59

 CNV, choroidal neovascularisation; DA, number of disc areas; ETDRS, early treatment diabetic retinopathy study; SD, standard deviation.

*  Standard deviation calculated using 95% confidence intervals from the trial report: SD = (upper limit − lower limit) / 3.92 · n for n≥60; SD = (upper limit − lower limit) / 4.128 · n for n≤60.

†  Point above which for every change in visual acuity a proportion of the treated patients achieve their change due to the treatment.

Table 3. 

 

Assumptions for Calculating Cutoff Points for Deciding Whether to Continue or Stop a Treatment

Study	Outcome	No. in Treated/ Reference Group	Follow-up	Change in Outcome, Mean (SD)
				Treated Group	Reference Group
Dubiner et al.5	IOP reduction (%)	21/21	Day 29, at 12 noon	30.1 (12.4)*	2.1 (16.5)*
Dehghan et al.6	Visual acuity (LogMar)	43/45	2 months	0.13 (0.27)*	0.02 (0.26)*
CITT7	NPC break reduction (cm)	60/54	12 weeks	10.4 (5.07)†	3.9 (4.98)†

 CITT, Convergence Insufficiency Treatment Trial; IOP, intraocular pressure; NPC, near point of convergence; SD, standard deviation.

*  Standard deviation calculated using standard errors (SE) from the trial report: SD = SE· n .

†  Standard deviation calculated using 95% confidence intervals from the trial report: SD = (upper limit − lower limit) / 4.128 · n .

Several observations can be made when plotting the curves of the normal distributions, or probability density functions (see Fig. 1). At the intersection of the curves, the probability densities in both the treated group and the nontreated (placebo) group are equal:  

 

For patients ending at the corresponding change in outcome, x, the probability is therefore zero that this change is due to the treatment. This change in outcome can be calculated by solving for x the following quadratic equation which results from equation 1:  

 

where  

 

For the first example, we derived data from the Minimally Classic/Occult Trial of the Anti-VEGF Antibody Ranibizumab in the Treatment of Neovascular AMD (MARINA) trial report.¹ In this trial, participants had AMD with either minimally classic or occult (with no classic lesions) choroidal neovascularization (CNV). They were treated with monthly intravitreal injections of ranibizumab or sham injections. We applied the calculations to the change in Early Treatment Diabetic Retinopathy Study (ETDRS) visual acuity. In this example (Fig. 1), at intersection x, the number of patients who achieved the corresponding change in visual acuity x due to the treatment with ranibizumab is zero. This change in outcome (visual acuity) is our cutoff point of interest. As illustrated in Figure 1, for a change in visual acuity above cutoff point x, for example, “A” or “B,” the probability density in the treated group is greater than in the placebo group, that is, there are more treated patients with that change than there are nontreated patients with the same change.  

Figure 1. 

 

Normal distribution models of change in visual acuity in sham and ranibizumab group at follow-up = 24 months in the MARINA trial. (A) Patients who gained 10 to 11 letters; (B) patients who gained 3 to 4 letters; x , patients who lost 5 letters.

The proportion of treated patients who ended above cutoff point x is the area under the distribution of the treatment group results above x: A_t, which is calculated using the error function:  

 

The proportion of nontreated patients who ended above cutoff point x is the area under the distribution of the placebo group results above x: A_r, which is calculated using the error function:  

 

The treatment-attributed effect (TAE), that is, the proportion of treated patients who ended above the cutoff point due to the treatment, is calculated using A_t and A_r:  

 

For 1, 3, 6, 12, and 24 months of follow-up in the MARINA trial we calculated cutoff point x and TAE in the change in visual acuity. Furthermore, we have applied the method to MARINA subgroup analysis results by Boyer et al.⁴ This allowed us to assess whether the method yields different results when addressing effect modification. The subgroup analyses were based on the 24-month visual acuity results segregated by age, initial visual acuity, CNV lesion size, or CNV lesion type. 

The second example was based on results of a trial of bimatoprost versus placebo by DuBiner et al. ⁵ We calculated cutoff point x and TAE in the percentage reduction of IOP after 29 days for treatment with bimatoprost for patients with elevated IOP. The third example was based on results of a trial of triamcinolone versus placebo by Dehghan et al. ⁶ We calculated cutoff point x and TAE in the change in LogMar visual acuity after 2 months in the treatment of refractory diabetic macular edema. The fourth example was based on results of a trial of office-based vergence/accommodative therapy with home reinforcement versus, among others, office-based placebo therapy with home reinforcement by the Convergence Insufficiency Treatment Trial Study Group. ⁷ We calculated cutoff point x and TAE in the 12-week reduction in centimeters of the NPC with vergence/accommodative therapy. 

Results

We explain the method with some example calculations (Fig. 1) based on the MARINA trial results at 24 months. First, we consider the modeled patient group “A” in Figure 1. This group includes all patients who gained 10 to 11 letters of visual acuity. These patients are represented by areas under the curve in both the intervention and the sham treatment group. The shaded area under the curves for the intervention group (the light area plus the dark area) represents the proportion of all treated patients who gained 10 to 11 letters: 2.3%. The area under the curve in the sham group (the dark shaded area) is the proportion of all sham patients who also gained 10 to 11 letters: 0.85%. The proportion of patients in the treated group gaining 10 to 11 letters who actually gained that visual acuity due to the treatment (the light shaded area) was (2.3 − 0.85)/2.3 * 100% = 63%. Group “B” in Figure 1 contains all modeled patients with a more modest gain of 3 to 4 letters visual acuity. If we do the same calculations for this group, the proportion of patients who gained 3 to 4 letters attributable to the treatment is smaller than that in group A: 42%. At the intersection of the curves, cutoff point x, this proportion is reduced to zero; in this example, the cutoff point is −5 (point x in Fig. 1). At this point, the proportion of patients who achieved this change was equal in both the intervention group and the sham group. The proportion of patients in the treatment group who lost 5 letters attributable to the treatment is therefore zero. This −5 value is our cutoff point at 24 months follow-up. As illustrated with groups A and B, for every value of the change in outcome above this cutoff point (less loss or more gain), there are patients who have achieved the change as a result of the treatment. The more a change in outcome lies above the cutoff point, the larger the proportion of treated patients who achieved this change as a result of the treatment. These patients would not have achieved this outcome if the treatment had not been given. 

Table 1 shows the results for five times of follow-up in the MARINA trial report. It shows the mean value and SD in the change in visual acuity derived from the trial article. It also displays the corresponding cutoff point x. Furthermore, it shows the proportion of treated patients who ended above this point, A_t , as well as the proportion of treated patients who ended above this point due to the treatment, TAE. Table 2 shows cutoff point x and TAE, over different levels of effect modifiers in the MARINA trial: age, initial visual acuity, lesion size, and lesion type. The means and SDs used in the calculations are specified. 

The means and SDs for calculating the cutoff points for the following examples are shown in Table 3. For IOP lowering with bimatoprost after 29 days, the cutoff point lies at an IOP reduction of 16% (see Fig. 2A). For treating refractory diabetic macular edema with intravitreal triamcinolone, after 2 months, the cutoff point lies at approximately 0.06 LogMar units visual acuity improvement (Fig. 2B). After treatment of convergence insufficiency with vergence/accommodation therapy, the cutoff point lies at approximately 7 cm reduction of the NPC (Fig. 2C). Figure 2 shows that the proportion of treated patients who achieved a change in the outcome better than the cutoff point attributable to the treatment is greatest in the bimatoprost trial report data, followed by the vergence/accommodation therapy trial and the triamcinolone report data. 

Figure 2. 

 

Normal distribution models of clinical trial results: Patients with high IOP (A), refractory diabetic macular edema (B), and convergence insufficiency (C). The intersections of the treated-groupcurve and the placebo curves are visible. The proportions of treated patients achieving a better change in the outcome than the cutoff point attributable to the treatment are depicted as shaded areas under the curves, with the size of the proportions indicated as a percentage (treatment-attributed effect, TAE).

Discussion

In this article we have analyzed the change in continuous outcome variables used in the follow-up of chronic disease during a treatment. We developed a new, straightforward method to calculate cutoff points above which, for every level of change, it is certain that some proportion of the treated patients achieved that change due to the treatment. The AMD example illustrates several relevant points regarding this method. We discuss these points in the following sections. 

This Method Produces a Cutoff Point That Limits the Range of Clinically Sound Cutoff Points

Clinicians may seek a cutoff point to decide to continue or stop a treatment for patients in their practice. It is important that using such a cutoff point does not lead to stopping the treatment in patients in whom the treatment is effective. As illustrated in the AMD example, the cutoff point at the intersection of the curves is the point at which for every smaller decline or greater improvement in visual acuity, a proportion of the treated patients has achieved the change attributable to the treatment. Stopping the treatment is therefore not appropriate in patients with a change better than the cutoff point at the intersection. The choice of a cutoff point used in clinical practice is therefore limited as it should not be located above the cutoff point calculated by this method. 

The Value of the Cutoff Point Depends on the Size and the Distribution of the Treatment Effect in the Research Population

The mean change in the outcome defines the position of the “treatment curve” such as depicted in Figure 1. The SD of the change in the outcome defines the width of the curve. The intersection of the treatment curve with the sham curve is dependent on the shape and position of the treatment curve (and the placebo curve). The cutoff point occurring at the intersection of the curves is therefore dependent on the efficacy of the treatment and its distribution width. 

A Limited Decline in Visual Acuity Can Reflect a Beneficial Effect of Therapy

As we show in the AMD example, the cutoff point may not be equal to a zero change in outcome. At 24 months of follow-up, a decline of 5 letters is the cutoff point. This implies that some patients may show a loss (of up to 5 letters) in visual acuity but nevertheless be better off than would have been the case if they had not been treated. In other words, the treatment has limited the deterioration of visual acuity. Intuitively, and in daily practice, “no change” is often interpreted as “no effect.” Our example shows that such an interpretation may not be in accordance with the evidence. 

The Cutoff Point May Change over Time

Table 1 shows that the cutoff point changes over time in the AMD example. This is because visual acuity in the sham group, and, initially, in the treated group, is not stable but alters over time. It is important to know the natural history of the outcome and the time period for the effect of the treatment to be at a maximum. For example, in IOP therapy this occurs fairly soon, while for the visual acuity change in AMD, it takes longer. The example of AMD shows that, even after a short time period, a cutoff value can be given such that above this value one can be certain that this change of visual acuity is achieved by the intervention. Changing or stopping treatment above the cutoff point may not be the best choice. Below this value, one can consider stopping or changing the therapy or wait for the development of the visual acuity and act according to the new cutoff value for that time period. Of course, other considerations for changing or stopping a treatment should be taken into account as well. 

The Cutoff Point May Differ across Levels of Effect Modifiers

The presented method can be of more practical value when subgroup analyses have been conducted so that the method can be applied per subgroup. In that case, a more patient-specific cutoff point can be selected for use in clinical practice. Table 2 shows that the cutoff point is different among groups with different age, initial visual acuity, CNV lesion size, and CNV lesion type in the AMD example, based on subgroup analysis results. In older patients, in patients with a good initial visual acuity, and in patients with large lesions, a loss of visual acuity of up to around 10 letters can still be an effect of the treatment in a proportion of patients. In occult lesions, the visual acuity loss may be more severe than in minimally classic lesions and still be an effect of the treatment in a proportion of patients. 

Application to Other Scales and Conditions

The method described here can be applied to scales and conditions other than the change in visual acuity in AMD. In addition, the method may be even more appropriate in the case of a continuous variable that is altered by a treatment but that has a stable natural course over a (relatively short) period of time. Examples of conditions with such outcomes are elevated IOP and convergence insufficiency. For IOP lowering with bimatoprost after 29 days as presented by DuBiner et al., the cutoff point can be calculated to be an IOP reduction of 16%. This means that, for a reduction in the IOP of more than 16% after 29 days, a proportion of treated patients will achieve that lowering because of treatment with bimatoprost. The European Glaucoma Society states that if the first choice monotherapy is not effective on IOP, it is preferable to switch to another agent. “Treatment is considered ‘effective' on the IOP when the observed IOP lowering effect on the treated patient is comparable to the published average effect of the same compound on a similar population.” ⁸ It is unclear what evidence this statement is based on. Based on meta-analysis, the published average effect of bimatoprost is ≥27% IOP reduction. ⁹ If 27% IOP reduction is assumed as a cutoff point in clinical practice, instead of 16% as determined from the method example, this means that the treatment will be stopped in patients in whom bimatoprost treatment was effective in reducing IOP. However, the clinical appropriateness of the method is also influenced by the implications involved in withholding a treatment. For instance, the decision to discontinue a treatment for high IOP or hypertension can be made relatively easy since there are several other treatments to choose from. 

Application of the method to the effect of intravitreal triamcinolone on diabetic macular edema results in a cutoff point close to zero change. The cutoff point of −0.06 (an improvement in visual acuity) suggests that, at 2 months follow-up, a small improvement needs to be observed in order to decide that the treatment has been effective. In this example, “no change” seems to imply “no effect,” with a value better than “no change” being the cutoff point. The convergence insufficiency example shows that, according to the results in the trial report used, an improvement of the NPC may occur in an ineffective treatment. Only a reduction of the NPC of more than 7 cm indicates that patients clearly achieved this reduction due to the treatment. 

In order to apply the method as presented here, input data should be based on RCT data. In such studies, a group that does not receive the treatment, with participants randomly assigned to this arm, should be included in order to obtain an unconfounded estimate of the change in the outcome variable without treatment. It is possible to also employ the method to assess the usefulness of switching from one treatment to another. Some caution is warranted in this case. When there are small differences between the effects of the treatments, with relatively large SDs, then the TAE will be relatively small above the default cutoff point or above any other cutoff point. This implies that it is difficult to attribute a change in outcome measure, after a change in treatment, to the treatment change in this situation. 

A way of proving that the choice of a cutoff point is appropriate would be to rerandomize those patients below the cutoff point (in whom the treatment is less likely to be effective) either to stop or to continue treatment and to compare the outcome in the two new groups. If further treatment would be ineffective in these patients, the outcome would not differ significantly between the groups, and stopping the treatment below the cutoff point would have been warranted at the moment of rerandomization. There may be a group of apparent nonresponders who are actually responders with an aggressive natural course that the treatment cannot counteract. In the patients selected for the trial, one half will show the natural (aggressive) course and the other half the course in case treatment is continued, and the proposed trial would show whether continuing treatment in the apparent “nonresponding” patients would have been warranted. Regression to the mean may be an issue here, as it will cause the last measurement before rerandomization to be closer to its real value than the initial measurement. This may add to the observation that the change in visual acuity after rerandomization is less dramatic than the change before rerandomization, with the treatment appearing less effective in the rerandomized population. 

Using cutoff points calculated by this method may also lead to the design of safer stopping trials, in which the treatment is (randomly) stopped or continued only in those patients below the calculated cutoff point instead of randomizing all patients to either stopping or continuing treatment. It can be expected that this will prevent the reoccurrence or exacerbation of the treated disease to a greater extent in those whose treatment was discontinued compared with randomizing all patients. ¹⁰ If we were to hypothetically rerandomize the MARINA trial patients at 24 months, 25% of the treated patients would be rerandomized, since the proportion of treated patients who ended above cutoff point was 75% (see Table 1). 

An issue remains regarding the variability of the estimate of the cutoff point. One can expect that the value will differ between similar RCTs. This would be a phenomenon comparable to the variability of the effect estimates between similar RCTs, for example, the difference in the relative risks or the difference in continuous outcomes. Further research could focus on developing methods to calculate the statistical variance of the cutoff point and to combine the estimates based on several RCTs. 

Furthermore, in employing a cutoff point in clinical practice, one must assess whether the measured change in the outcome is likely to be real. When a clinician suspects that the observed change in outcome is due to measurement error (frequently the case in single IOP measurements) then basing decision making on that particular set of measurements would not be warranted. Considering repeating the measurement should be an issue in every assessment of treatment effect. 

For the examples, we used the mean and SD derived from the trial reports. Ideally, the original data from individual patients should have been used since it is likely that the distributions are skewed, but we did not have access to these data. These results should therefore only be used as an illustration of the method and should not be used as the actual cutoff points for stopping or continuing ranibizumab treatment for AMD, and similarly for the other treatments and conditions mentioned. Regarding AMD, one should also keep in mind that follow-up decisions are not solely based on the change in one outcome, such as visual acuity. For instance, decision-making may also be based on the finding of unresolved macular edema on optical coherence tomography. However, again, this may raise the question of how much a change in the outcome—macular edema—reflects the effect of treatment. 

Our method can be readily applied to other trial data. In the monitoring of any chronic (ophthalmic) disease, appropriate evidence-based cutoff points are of great value to clinicians. Such results will contribute to evidence-based monitoring, which has not been studied extensively. ^11,12 It is possible that a similar method has been described before, but we are unaware of such a publication. We recommend using this method in future RCTs in which a change in an outcome versus baseline is being studied, for example, visual acuity, IOP, or flare (in uveitis). We recommend presenting the calculated cutoff point as well as the proportion of treated patients ending above this cutoff point. 

Acknowledgments

The authors thank the reviewers for their help in further defining the method and addressing important issues in its interpretation. 

Footnotes

 Supported by The Netherlands Organization for Health Research and Development (ZonMw), The Hague, Grant 152001002.

Footnotes

 Disclosure: M. Elshout, None; M.I. van der Reis, None; C.A.B. Webers, None; E.C. La Heij, None; F. Hendrikse, None; J.S.A.G. Schouten, None