Evaluation of Sampling Strategies for Modeling Survival of Uveal Malignant Melanoma

Abstract

purpose. To evaluate sampling strategies used to estimate survival after uveal malignant melanoma that exclude some patients who would be censored from the analysis.

methods. Simulation was performed on a population-based data set of 133 patients who had an eye enucleated because of uveal melanoma. One thousand bootstrap samples of 80 patients were drawn, without replacement, according to three sampling strategies: a random draw (conventional strategy), a draw limited to patients who died of the tumor or survived at least 10 years without metastasis (“late-censoring” strategy), and a draw modified so that 40 patients died of melanoma and others survived at least 10 years without metastasis (“fifty-fifty” strategy). The bias in the Kaplan-Meier analysis and Cox proportional hazards regression was quantified.

results. The late-censoring strategy decreased the proportion of censored patients from 53% to 42%, whereas the fifty-fifty strategy assigned 50% of patients to this group. The former strategy overestimated mortality, the excess being 5.2% and 3.7% at 10 and 20 years, respectively. The latter strategy underestimated mortality, the bias being 1.6% and 4.6% at 10 and 20 years, respectively. The bias differed according to categories of explanatory variables so that the log-rank test statistic was inflated a median of 1.08 times (range, 0.73–1.87) and 1.14 times (range, 0.87–1.84), and the Wald χ² statistic of the Cox regression was inflated a median of 1.18 times (range, 0.79–2.13) and 1.16 times (range, 0.71–2.02), respectively, when the late-censoring and fifty-fifty strategies were applied.

conclusions. Sampling strategies that exclude on purpose a proportion of patients who would be censored produce biased statistics, because they violate assumptions of survival analysis. Only random sampling from an underlying population produces unbiased survival estimates.

Recently, novel ways of sampling for Kaplan-Meier and Cox proportional hazards regression survival analyses have been adopted in several studies. ^{1 2 3 4 5 6 7} In these studies, the data sets were designed to include an unequal ⁵ or an equal ^{1 2 3 4 6 7} number of patients who either died of uveal melanoma or survived at least for a specified time after treatment. Patients who would have been censored for any reason before the specified time, typically 10 to 15 years after treatment, were excluded. ^{4 5 6 7} The data were analyzed according to standard procedures. Usually these studies did not comment on the use of unconventional sampling strategies. ^{1 2 4 5 6 7}  

We used simulation to assess the bias that could be introduced by unconventional sampling into analysis of the mortality rate in uveal melanoma. 

Patients and Methods

Population Data Set

A data set used to study the association between microvessels and survival after enucleation for uveal melanoma ^{8 9} was reanalyzed to asses the effect on Kaplan-Meier and Cox regression analysis of different sampling strategies. This population-based set consists of 167 consecutive patients treated from 1972 to 1981 when enucleation was the only treatment for all but the smallest uveal melanomas. ⁸ Median follow-up of survivors was 17 years (range, 0.2–26.9). 

Independent Variables

A selection of categorical and continuous variables was used that were or were not significantly associated with melanoma-specific survival in the population data set. ^{8 9} The variables chosen were gender; age at enucleation (divided in tertiles for Kaplan-Meier analysis); involvement of ciliary body by the tumor (not involved, involved); height and largest basal diameter of the tumor (LBD; divided in three categories for Kaplan-Meier analysis); presence of epithelioid cells (absent, present); grade of pigmentation (weak, strong); presence of microvascular loops and networks, consisting of at least three back-to-back loops, ¹⁰ analyzed as an ordered categorical variable that considers networks to be an advanced stage of loops ⁸ (no loops, loops without networks, loops forming networks); and microvascular density (MVD) obtained from the most highly vascularized area (hot spot) of the tumor (divided in quartiles for Kaplan-Meier analysis and square-root transformed for Cox regression). ^{9 11 12}  

Bootstrapped Data Sets

Simulation was limited to 133 (80%) patients for whom there was complete data. Three bootstrap samples consisting of 1000 replications of data from 80 patients were drawn at random with replacement. The first set consisted of replications drawn regardless of outcome and follow-up time (conventional strategy). The second set consisted of replications drawn from a reduced sample of 108 patients, which excluded 25 patients who were lost to follow-up or had died of causes other than uveal melanoma within the first 10 years after enucleation (“late-censoring” strategy). The third set was also drawn from the reduced data set, but the bootstrapping algorithm was modified so that 40 (50%) patients in each replication had died of uveal melanoma and 40 (50%) had survived for 10 years or more (“fifty-fifty” strategy). 

Statistical Methods

The data sets were constructed and the data were analyzed on computer (Stata statistical software, ver. 7.0; Stata Co., College Station, TX). 

The number of events and censored patients during the early and late periods (before versus 10 years after enucleation) for each strategy were calculated as the mean of all replications. Survivor function was analyzed using the Kaplan-Meier product-limit method. ¹³ An average survival curve representing each sampling strategy was obtained by plotting the survival for all 80,000 patients in each bootstrapped set. It can be shown that this curve closely approximates the survival curve that would be obtained by calculating point by point the average of the 1000 individual survival curves. Patients judged to die of causes unrelated to uveal melanoma were censored at the time of death. Confidence intervals were calculated according to Greenwood. ¹⁴ Empiric confidence limits were obtained by plotting the individual survival curves for the first 100 replications in each bootstrapped set. 

Survival curves by categories of each independent variable were constructed to assess bias in estimating survival proportions, as described for the entire data set. Categories were compared with the log-rank test. ¹³ Bias related to hypothesis testing was quantified by calculating the ratio of the average log-rank test χ² statistic, obtained as the mean of all replications in each set, to that of the conventional strategy. The probability corresponding to the average χ² statistic was obtained by computer (StaTable, ver. 1.0.1; Cytel Co., Cambridge, MA). 

Cox proportional hazards regression was used to calculate the average hazard ratio (HR) and average Wald χ² statistic ^{13 15} for each independent variable, obtained as the mean of all replications in each set. In addition, a previously derived multivariate model was fitted. ⁹ Bias related to HR was quantified by calculating the ratio of the corresponding HR estimate to the estimate obtained from data on the 133 patients in the starting data set. Bias related to hypothesis testing was quantified by calculating the ratio of the corresponding Wald χ² statistic to that obtained by the conventional strategy. 

Results

Events and Censoring

Of the 133 patients, 47% died of uveal melanoma and 53% were censored (Table 1) . Both proportions were similar when sampling was conventional. The late-censoring strategy increased the proportion of tumor deaths to 58% and decreased that of censored patients to 42%. By definition, sampling by the fifty-fifty strategy assigned 50% of patients to both groups. 

Of patients observed for less than 10 years (early period) and longer (late period) after enucleation, 34% and 76% were censored, respectively (Table 1) . Thus, 25 (36%) of the 70 censored patients were censored during the early period. The proportions were similar with conventional sampling. By definition, no early-period censoring occurred with unconventional sampling. The proportion of patients censored during the late period was not affected by late-censoring strategy, whereas the fifty-fifty strategy increased this proportion to 82% (Table 1) . 

The proportions of events and censored observations predicted by the simulation were consistent with those reported in published studies in which unconventional sampling was applied (Table 1) . 

Survivorship Function

The Kaplan-Meier curve obtained by conventional sampling (Fig. 1A) was effectively indistinguishable from the observed survival of the 133 patients in the starting data set. The empiric bootstrapped confidence limits agreed with the 95% confidence interval calculated with the Greenwood method (Fig. 1A) . 

The plot obtained by the late-censoring strategy progressively overestimated mortality during the early period compared with the conventional strategy, because the number of patients to be censored was deflated (33 vs. 42) and that of events inflated (36 vs. 29) during this period (Table 1) . The curve dropped proportionally more with each event, and excess mortality was 5.1% (survival, 67.5% vs. 72.6%) by 5 years and 5.2% (54.4% vs. 59.6%) by 10 years (Fig. 1B) . Bias decreased when the number of patients to be censored became inflated toward the start of the late period (Table 1 ; 33 vs. 27), and mortality excess decreased to 4.5% (survival, 47.8% vs. 52.3%) by 15 years and to 3.7% (41.8% vs. 45.5%) by 20 years (Fig. 1B) . The bootstrapped confidence limits agreed with the Greenwood formula (Fig. 1B) . 

The plot obtained by the fifty-fifty strategy initially slightly overestimated mortality, because fewer patients to be censored were enrolled than with the conventional design (50 vs. 52) and a few events more (31 vs. 29) occurred during the early period (Table 1) . Excess mortality was 0.4% (survival, 72.2% vs. 72.6%) by 5 years (Fig. 1C) . The number of patients to be censored soon became inflated, however, reaching a maximum by the start of the late period (Table 1 ; 40 vs. 27, respectively). A smaller decrease occurred with each subsequent event, resulting in progressive underestimation of mortality. Bias was 1.6% (survival, 61.2% vs. 59.6%) by 10 years, 3.0% (55.3% vs. 52.3%) by 15 years, and 4.6% (50.1% vs. 45.5%) by 20 years (Fig. 1C) . The empiric confidence interval converged toward 50% survival, dictated by the sampling design, and the Greenwood formula was invalid (Fig. 1C) . 

Effect Size and Hypothesis Testing in Kaplan-Meier Analysis

Survival curves by categories of independent variables (Fig. 2) revealed that the excess mortality related to the late-censoring strategy was more pronounced in categories with poor survival (in which early deaths predominated and fewer patients remained to be censored late), whereas the mortality underestimate related to the fifty-fifty strategy was more pronounced in categories with good survival (in which early deaths were rare and many patients remained to be censored late). In both instances, the difference between categories was inflated (Figs. 2A 2B 2C 2D) . If the curves crossed, the difference between categories could decrease, however (Figs. 2E 2F 2G) . 

The abnormal shape of confidence limits in the fifty-fifty strategy did not apply to curves by independent variable category, because survival of patients in individual categories did not converge toward any predetermined cumulative proportion. 

The late-censoring strategy inflated the χ² statistic of the log-rank test by a median of 1.08 times (range, 0.73–1.87), and the fifty-fifty strategy inflated it by a median of 1.14 times (range, 0.87–1.84), with corresponding undue improvement in the probability in both cases (Table 2) . 

Effect Size and Hypothesis Testing in Cox Regression

In univariate Cox regression, the hazard ratio (HR) estimated by all strategies was larger than the population estimate derived from data of all 133 patients, except when the survival curves crossed (Table 3) . The median ratio of HRs between estimates was 1.04, 1.02, and 1.06 for the conventional, late-censoring and fifty-fifty strategies, respectively. The HR estimated by the late-censoring strategy was smaller or larger compared with the conventional strategy (median ratio of HRs, 0.99), whereas the fifty-fifty strategy provided uniformly larger estimates, unless survival curves crossed one another (median ratio of HRs, 1.03). With the multivariate model, results were similar (Table 3) . 

In univariate Cox regression, the late-censoring strategy inflated the Wald χ² statistic by a median of 1.18 times (range, 0.79–2.13), and the fifty-fifty strategy inflated it by a median of 1.16 times (range, 0.71–2.02), with corresponding undue improvement in the probability compared with the conventional strategy (Table 3) . With the multivariate model, the χ² statistics of explanatory variables were also generally inflated (Table 3) . 

Discussion

Conventional random sampling resulted in an unbiased estimate of the observed survival of the underlying population, as expected. Both unconventional sampling strategies distorted the survivorship function. The late-censoring strategy caused a steeper initial decline in the survival curve, whereas the fifty-fifty design underestimated late mortality and forced the curve to converge toward 50%. The distortion would be greater if observed mortality would deviate more from 50% than is the case with uveal melanoma. 

Based on the simulation, studies in which the late-censoring strategy was used have probably overestimated mortality by 4% to 5%. ⁵ The bias was greater for subgroups with high mortality, and statistical significance probably was inflated. Similarly, studies that have used the fifty-fifty sampling design ^{1 2 3 4 6 7} probably underestimated mortality by 3% to 5%. The bias for subgroups with low mortality was greater, which again inflated statistical significance to a variable extent. In addition, the fifty-fifty strategy is expected to increase the hazard ratio by a mean of 1.03 times. It is impossible to predict exact bias for individual covariates, however. 

The use of unconventional sampling is not always clearly mentioned. In a study that evaluated insulin-like growth factor receptor as a predictor of metastasis, 36 patients were analyzed, of whom 18 (50%) died of uveal melanoma, 3 died of other diseases, and 15 were alive at the end of the follow-up. ⁷ According to a table published in the report of that study, the follow-up times of those who died of tumor (1–150 months) and those who were censored from the analysis (186–245 months) were very disparate. ⁷ These observations uncover a fifty-fifty sampling strategy. 

Unconventional sampling is especially likely to lead to misinterpretation when the covariate is a weak predictor, sample size is small, or both. For example, the probable inflation of statistical significance associated with the fifty-fifty strategy casts doubt on the reported association between expression of insulin-like growth factor receptor and mortality in melanoma, which is of borderline significance (P = 0.035). ⁷ The corresponding χ² statistic is 4.45. In the simulation, survival curves that similarly did not cross each other yielded χ² values that were 1.07 to 1.84 times larger than those obtained by conventional sampling. The correct χ² value consequently was probably between 2.42 and 4.16, corresponding to a true probability of 0.13 to 0.041. 

In general, investigators who have used unorthodox sampling have not mentioned why these strategies were adopted, ^{1 2 4 5 6 7} except once when they thought that variables associated with prognosis would be easier to spot and multiple covariates easier to compare. ³ One might presume that they wanted to conduct a case–control type of study, which would be proper if the data were analyzed by logistic regression, ¹⁶ a method that has been appropriately used to model survival in uveal melanoma. ¹⁷ Another reason may be that a new assay was being tested on previously collected specimens and there were not enough resources to run the assay on every one. 

Even though logistic regression is a proper way to compare survival by covariates between patients who had or had not died of melanoma by a given time point, it answers a different question than regression based on survival data. ¹⁸ Logistic regression returns the log odds ratio of dying at a specified time point. Cox regression yields the relative hazard of death, which is assumed to be constant over time, unless time-dependent covariates are included in the model. ^{15 18} Cox regression makes more efficient use of time-to-event data than does logistic regression, which disregards both data of patients who would be censored before the time point analyzed and the subsequent survival experience of patients who were alive at that point. Logistic regression is also indifferent to the exact time when death occurred. Survival analysis remains the most valid and the most efficient method to analyze time-to-event data. 

A properly constructed Kaplan-Meier survival curve provides an unbiased estimate of the probability of survival, even in the extreme case of a single patient. ¹⁹ Proper construction depends on three factors: correct recording of the time of entry, recording of the time of death or censoring, and the assumption that the chance of being censored is unrelated to the risk of dying. ^{18 19} The survival curve is calculated on the basis of patients who are at risk of dying on each successive day. ^{18 19} Censoring occurs when the time to death is unknown because of termination of the study, loss to follow-up, or withdrawal for other reasons. ¹⁸ Censored observation should be incomplete only because of random factors, so that, conditionally on the values of all explanatory variables, the prognosis for any patient who has survived a certain time should not be affected if he or she is censored. ^{15 20} In particular, censoring must be unrelated to future lifetime. ²¹ Whether a patient is included in a survival study in the first place must also be determined before knowing the outcome. ¹⁴  

These principles are violated in the late-censoring and fifty-fifty strategies. In both strategies, enrollment and censoring depend on future lifetime. Moreover, both make censored patients initially “immortal,” because they can die only during the late period. Only the patients who die of melanoma are truly at risk during the early period. The censored observations in these strategies bear resemblance to left-truncated data or delayed entry. ^{15 21} These factors also distort the Cox regression analysis. 

We hope that our simulation will aid those interested in survival statistics in general and mortality in uveal melanoma in particular and that it will help reviewers to recognize the unconventional sampling strategies described so that they can guide authors to use unbiased designs. Finally, we note that our treatise has not addressed the problem of competing risks in the analysis of melanoma-specific survival. ^{22 23}  

 Supported by the Helsinki University Central Hospital Research Fund Grant TYH1217 and the Finnish Medical Foundation, Helsinki, Finland.

 Submitted for publication December 27, 2002; revised February 9, 2003; accepted February 18, 2003.

 Disclosure: T. Kivelä, None; P. Grambsch, None

 The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked “advertisement” in accordance with 18 U.S.C. §1734 solely to indicate this fact.

 

Table 1.

 

Events and Censoring during Early and Late Period, According to Sampling Strategy in the Simulation and Published Data Sets When Analyzing Mortality in Malignant Uveal Melanoma

Data Set		Early Period*			Late Period*			Entire Sample
				Events	Censored	Events	Censored	Events	Censored
		N	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
Underlying population	133	49 (66)	25 (34)	14 (24)	45 (76)	63 (47)	70 (53)
Simulated data
Conventional strategy	80	29 (67)	14 (33)	9 (25)	27 (75)	38 (48)	42 (52)
Late-censoring strategy	80	36 (100)	0 (0)	11 (24)	33 (76)	47 (58)	33 (42)
Fifty-fifty strategy	80	31 (100)	0 (0)	9 (18)	40 (82)	40 (50)	40 (50)
Published data
Late-censoring strategy 5	132	N/R	N/R	N/R	N/R	73 (55)	59 (45)
Fifty-fifty strategy 7	36	16 (100)	0 (0), †	2 (10)	18 (90), †	18 (50)	18 (50)

Figure 1.

 

Kaplan-Meier estimate of melanoma-specific survival (thick black line) with the corresponding 95% confidence interval calculated by the Greenwood method (dotted black lines) according to (A) conventional, (B) late-censoring, and (C) fifty-fifty sampling strategies, together with empiric confidence limits obtained by plotting the first 100 bootstrapped survival curves (thin gray lines). Note that the late-censoring strategy overestimates mortality (B), especially in the earlier part of the curve, whereas the fifty-fifty strategy underestimates mortality in the latter part of the curve (C) compared with conventional sampling strategy (thin black line). The calculated confidence interval does not agree with the abnormally shaped bootstrapped limits when the fifty-fifty strategy is used (C).

Figure 2.

 

Kaplan-Meier estimate for melanoma-specific survival according to the presence of epithelioid cells (A, B), largest basal tumor diameter (C, D), gender (E, F), and presence of microvascular loops and networks (G, H) based on late-censoring (A, C, E, G) and fifty-fifty (B, D, F, H) sampling strategies (thick lines), compared with conventional sampling (thin lines). Note that the late-censoring strategy overestimates mortality more strongly when a category is associated with poor prognosis than when it is associated with good prognosis (A, C, G). The fifty-fifty strategy underestimates mortality more strongly when a category is associated with good prognosis, and it also overestimates early mortality when a category is associated with poor prognosis (B, D, H). As a consequence, the difference between categories is exaggerated by both strategies. If any two survival curves cross one another, the bias is often reversed and the difference between categories diminishes (E, F, and loops-only versus networks in G).

Table 2.

 

The Effect of Unconventional Sampling on Log-Rank Test in Analyzing Mortality in Malignant Uveal Melanoma

Explanatory Variable*	df	Conventional Strategy			Late-Censoring Strategy					Fifty-fifty Strategy
				χ2	P	χ2	Ratio, †	P	χ2	Ratio, †	P
Gender, ‡	1	1.64	0.20	1.20	0.73	0.27	1.42	0.87	0.23
Age at enucleation	2	6.16	0.046	11.53	1.87	0.0031	11.31	1.84	0.0035
Ciliary body involvement	1	9.64	0.0019	10.51	1.09	0.0012	11.71	1.21	0.00062
Tumor height	2	10.20	0.0061	14.52	1.42	0.00071	15.94	1.56	0.00035
Largest basal tumor diameter	2	14.18	0.00083	20.04	1.41	<0.00001	19.13	1.35	0.00007
Epithelioid cells	1	15.12	0.00052	16.46	1.09	0.00027	16.92	1.12	<0.00001
Grade of pigmentation	1	7.38	0.0066	7.98	1.08	0.0047	7.93	1.07	0.0049
Microvascular patterns, ‡	2	13.27	0.0013	15.31	1.15	0.00047	14.93	1.13	0.00057
Microvascular density	3	18.25	0.00039	19.47	1.07	0.00022	20.84	1.14	0.00011

Table 3.

 

The Effect of Unconventional Sampling Strategies on Effect Size and Hypothesis Testing in Cox Proportional Hazards Regression of Mortality in Malignant Uveal Melanoma

Explanatory Variable	Population Estimate Hazard Ratio	Conventional Strategy							Late-Censoring Strategy									Fifty-fifty Strategy
				Hazard Ratio			Wald Test			Hazard Ratio			Wald Test					Hazard Ratio			Wald Test
				Estimate	Ratio	χ2	P	Estimate	Ratio*	χ2	Ratio, †	P	Estimate	Ratio*	χ2	Ratio, †	P
Univariate analysis
Gender, ‡	1.322	1.407	1.06	1.53	0.22	1.237	0.94	1.21	0.79	0.27	1.245	0.94	1.09	0.71	0.30
Age at enucleation, §	1.027	1.027	1.00	4.71	0.03	1.039	1.01	10.04	2.13	0.0015	1.041	1.01	9.50	2.02	0.0021
Ciliary body involvement, ∥	2.532	2.719	1.07	8.19	0.0042	2.636	1.04	9.50	1.16	0.0021	2.862	1.13	9.72	1.19	0.0018
Tumor height, §	1.133	1.139	1.01	8.23	0.0041	1.183	1.04	14.00	1.70	0.00018	1.202	1.06	14.57	1.77	0.00014
Largest basal tumor diameter, §	1.127	1.136	1.01	8.75	0.0031	1.149	1.02	12.51	1.43	0.00040	1.150	1.02	11.40	1.30	0.00073
Epithelioid cells, ¶	3.089	3.389	1.10	12.63	0.00038	3.287	1.06	15.13	1.20	0.00010	3.504	1.13	14.51	1.15	0.00014
Grade of pigmentation, #	3.144	3.790	1.21	5.95	0.015	3.500	1.11	6.70	1.13	0.0096	3.890	1.24	6.37	1.07	0.012
Microvascular patterns, **	1.833	1.905	1.04	10.28	0.0013	1.861	1.02	12.05	1.18	0.00050	1.957	1.07	11.92	1.16	0.00056
Microvascular density, ††	1.338	1.360	1.02	12.04	0.00052	1.363	1.02	14.31	1.19	0.00016	1.390	1.04	14.18	1.18	0.00017
Multivariate Analysis
Largest basal tumor diameter, §	1.105	1.114	1.01	5.56	0.018	1.117	1.01	6.94	1.25	0.0084	1.115	1.01	6.07	1.09	0.014
Epithelioid cells, ¶	2.437	2.807	1.15	7.70	0.0055	2.627	1.08	8.73	1.13	0.0031	2.908	1.19	8.98	1.17	0.0027
Microvascular patterns, **	1.348	1.451	1.08	3.55	0.060	1.430	1.06	3.92	1.10	0.048	1.373	1.02	2.81	0.79	0.094
Microvascular density, ††	1.231	1.248	1.01	6.07	0.014	1.262	1.03	6.85	1.13	0.0089	1.281	1.04	6.56	1.08	0.010