External Validation of the Liverpool Uveal Melanoma Prognosticator Online

Abstract

Purpose: To validate the Liverpool Uveal Melanoma Prognosticator Online (LUMPO) in a cohort of patients treated at the University of California-San Francisco (UCSF).

Methods: A retrospective chart review was performed of 390 patients treated between 2002 and 2007 for choroidal melanoma at UCSF. Similar patients (n = 1175) treated at the Liverpool Ocular Oncology Centre (LOOC) were included in the study. The data were analyzed using the model previously developed for LUMPO, an online prognostication tool combining multiple prognostic factors. Main outcome measures included all-cause mortality and melanoma-specific mortality. Reliability of the survival estimates in each group of patients was indicated by the C-indices of discrimination and Hosmer-Lemeshow test.

Results: Patients treated at UCSF tended to be younger with thicker tumors, and were more likely to receive proton beam radiotherapy as primary treatment compared to patients at LOOC. There were no significance differences with respect to ciliary body involvement, melanoma cytomorphology, and mitotic counts between the two groups. Death occurred in 140/390 (35%) patients from UCSF and 409/1175 (34%) patients from LOOC, with no difference in overall mortality by Kaplan-Meier analysis (log rank test, P = 0.503). For all-cause mortality and melanoma-specific mortality, the C-index of discrimination and Hosmer-Lemeshow test at 5 years after treatment indicated good discrimination performance of the model, with no statistically significant difference between observed and predicted survival.

Conclusions: Despite differences between the two cohorts, external validation in patients treated at UCSF indicates that LUMPO estimated the all-cause and melanoma-specific mortality well.

Approximately 45% of patients with uveal melanoma will die from metastatic disease within a 15-year period after treatment of their primary tumor.¹ Uveal melanoma spreads hematogenously, with the most common site of metastasis being the liver. Metastases are rarely detectable at the time of primary ocular treatment and become evident months or years after apparent good health, which indicates the need for estimating if and when metastatic disease is likely to develop. Such prognostication, if sufficiently reliable, is reassuring for patients with minimal risk for metastasis.^2,3 Although systemic surveillance for metastases does not have a clear survival benefit,⁴ identifying patients at high risk for metastasis is important for clinical trial entry and any further treatments for metastatic disease that may become available in the future. There is also some evidence that quality of life is improved by prognostication, even when the survival probability is found to be poor.² 

There are many prognostic factors for metastatic death from uveal melanoma. Anatomic predictors, which form the basis of the American Joint Committee on Cancer (AJCC) Tumor, Node Metastasis (TNM) staging, include largest basal tumor diameter, tumor thickness, ciliary body involvement, and extraocular extension.^5–9 The TNM staging system is imprecise and is therefore applicable only to large groups of patients and not to individual patients.¹⁰ Histologic predictors include epithelioid melanoma cytomorphology, high mitotic count and certain extravascular matrix patterns, particularly closed loops, and tumor microvascular density.^11–15 Genetic predictors include partial or total chromosome 3 loss (“monosomy 3”), chromosome 8q gain, lack of chromosome 6p gain, and class 2 gene expression profile (GEP).^16–21 Uveal melanomas were also reported to fall into two classes based on GEP, with a 92-month survival probability of 95% in class 1 vs. 31% in class 2, and with nearly all metastatic deaths occurring among class 2 patients.¹⁷ However, it was later found that a small percentage of class 1 tumors also give rise to metastasis, and class 1 tumors were subsequently subdivided into class 1A (“low risk”) with 2% risk of metastasis, and class 1B (“intermediate risk”) with 21% risk of metastasis.²² A recent study found that that PRAME (preferably expressed antigen in melanoma) confers increased metastatic risk in class 1 or disomy 3 tumors.²³ 

Although genetic typing is highly accurate in determining which patients will develop metastatic disease, it does not predict when such disease is likely to develop. Survival time is better estimated by combining anatomic, histologic, and genetic predictors.²⁴ However, a problem with such multivariate analysis is bias caused by missing data. Bias also arises because of competing causes of death unrelated to the uveal melanoma.²⁵ Because these are censored in Kaplan-Meier analysis, this tends to exaggerate the apparent metastatic mortality, especially in elderly patients, who are more likely to die of other causes.^1,16,26 

The Liverpool Uveal Melanoma Prognosticator Online (LUMPO) is an online tool that was developed to estimate survival probability after treatment of choroidal melanoma.¹⁰ It combines clinical, histologic, and genetic data to enhance reliability of prognostication in individual patients and to estimate survival time as well as risk of metastasis. It minimizes bias from missing data by extrapolating from other predictors. To avoid bias from competing risks, LUMPO estimates relative survival using all-cause mortality in patients with choroidal melanoma and compares it to mortality from that of the general British population matched for age and sex using census data. This prognostic tool was developed and validated in 2012 with data on 3653 British patients, with histologic data in 1778 patients and genetic data in 738. A total of 1235 patients had died, and the model was calibrated using all-cause mortality.¹⁰ 

The LUMPO consists of two models, one of which uses only anatomic data (the “clinical” model) and the other also including histologic and genetic data (the “laboratory” model). These models were validated internally at the Liverpool Ocular Oncology Centre (LOOC) by bootstrapping.¹⁰ Bootstrapping is a method commonly used for internal validation in which a bootstrap sample is selected from the original sample where each subject has equal probability of being selected, and each can be selected more than once. This is repeated several hundred times, and the variance of statistics is analyzed for internal validation of a prediction model. However, to our knowledge, LUMPO has not previously been validated externally at another institution, and it is not known whether it is generalizable to patients treated elsewhere. 

The aim of our study was to validate LUMPO in a cohort of patients treated at the University of California-San Francisco (UCSF). 

Methods

A retrospective chart review was performed on all patients treated for choroidal melanoma at UCSF between 2002 and 2007. Patients were excluded if they had bilateral choroidal melanoma, or if there was missing information regarding baseline tumor diameter and tumor thickness. Patients were also excluded if they were known to have metastases at the time of diagnosis or if they did not receive treatment. 

Patient demographics included age and sex. Anatomic data collected included ciliary body involvement (i.e., pars plicata and pars plana), extraocular spread, and ultrasonographic measurements of largest basal tumor diameter and thickness. Histologic information included the presence or absence of epithelioid cells and mitotic cell count (number of mitoses per 40 high-powered fields). Presence or absence of closed extravascular matrix loops was not available in the pathology reports from UCSF during this time period. None of the patients in the UCSF dataset had genetic data available because tumor biopsy and prognostic genetic typing had not yet been adopted. 

Survival data were gathered from the UCSF Cancer Registry, which includes all cancer patients diagnosed and/or treated at UCSF. Patient vital status was updated at least annually through readmissions to the hospital, contact with primary care physicians, Department of Motor Vehicles records, contact with the patient's family, and information from the state registry. The UCSF Cancer Registry is required to update survival status at least every 15 months in a minimum of 80% of patients. Survival data for this study were obtained in 2014. The 2002–2007 time period was selected to allow for greater than 5 years of follow-up for all patients, and adequate time to gather sufficient survival information through these multiple methods. Due to the difficulty of obtaining paper charts from before 2002, this was selected as the starting point. 

Data were obtained from the LOOC database using the same inclusion and exclusion criteria. The Liverpool Ocular Oncology Centre notified the U.K. National Health Service Cancer Registry of all patients with ocular melanoma at the time of their initial diagnosis. Survival information was then collected from the Cancer Registry, which automatically informed LOOC of the date and certified cause of death of all patients residing in mainland Britain (i.e., Scotland, England, and Wales but not Northern Ireland). Genetic data were available at LOOC beginning in 1999 with fluorescence in situ hybridization (FISH) performed on all patients undergoing local resection or enucleation, and on all consenting patients from 2007 using multiplex ligation-dependent probe amplification (MLPA) and/or microsatellite analysis (MSA). 

The LUMPO model has been described by Eleuteri et al.¹⁰ In brief, an accelerated failure time model was used,²⁷ and missing data points were estimated using the alternating conditional expectations algorithm,^28,29 which predicts each missing data point as a function of the other available pieces of information. If histologic or genetic data were available, the laboratory model was used. If both histologic and genetic information were not available, LUMPO would default to the clinical model. 

Demographics and tumor characteristics of patients treated at LOOC were compared with those of UCSF by Fisher's exact test and Wilcoxon rank sum test. Survival of patients at LOOC and UCSF were compared using Kaplan-Meier analysis and log rank test. The performance of the model was assessed in two ways: discrimination using the Harrell C-index^30,31 and calibration using the Hosmer-Lemeshow test,^32,33 which are commonly used to assess prediction performance in survival analysis and to test goodness of fit in logistic regression, respectively. The Harrell C-index of discrimination measures the separation of two survival distributions, and any value of 0.7 or above within the 95% confidence interval indicates good performance. The Hosmer-Lemeshow test is a goodness-of-fit test for logistic regression. For this test, the null hypothesis is that there is no difference between observed and predicted mortality, so a P value greater than 0.05 indicates that observed survival is not significantly different from predicted mortality.^30–33 All-cause and melanoma-specific mortality were analyzed, where melanoma-specific mortality was obtained by subtracting patient survival from control survival. 

The Institutional Review Board of UCSF prospectively granted approval for this study (No. 13-11313), which followed the tenets of the Declaration of Helsinki. 

Results

Comparison Between UCSF and LOOC Cohorts

A total of 573 patients were treated for choroidal melanoma at UCSF between 2002 and 2007. The study included 390 patients (68%) after excluding 178 patients because of missing tumor dimensions (31%), 2 patients because of nontreatment (0.3%), and 3 patients due to systemic metastases at the time of diagnosis (0.5%). During the same time period, 1175 similar patients were treated at LOOC. 

The Table compares the baseline characteristics between the two patient groups. The sex ratio was the same in both populations, but those treated at UCSF tended to be younger. Compared to LOOC, tumors treated at UCSF were apparently thicker, but fewer were found to have extended extraocularly. The prevalence of ciliary body involvement was similar in the two groups. Compared to LOOC patients, those treated at UCSF were more likely to receive proton beam radiotherapy and less likely to undergo enucleation or other forms of treatment. At UCSF, epithelioid cells were identified in 65 out of 125 (52%) patients in whom melanoma cytomorphology was known, mostly as a result of fine-needle aspiration biopsy. At LOOC, epithelioid cells were documented in 371 out of 581 (64%) tumors examined histologically, many of which had been treated by enucleation or local resection (P < 0.05). The mitotic counts were similar in the two groups. Presence or absence of closed extravascular matrix loops was not included in the pathology reports during this time period at UCSF. The median follow-up times, defined as date of treatment to date of death for patients who had died, and date of treatment to end of study date (December 1, 2014) for patients who were alive, were 8.0 (range, 0.1–12.8) years and 7.7 (range, 0.1–12.9) years for patients treated at UCSF and LOOC, respectively, with no statistically significant difference detected (Mann-Whitney U test, P = 0.743). Death occurred in 140/390 (35%) patients from UCSF and 409/1175 (34%) patients from LOOC. Kaplan-Meier analysis revealed that overall mortality was the same between the two groups (log rank test, P = 0.503, Fig. 1). 

Table

 

Baseline Characteristics and Primary Ocular Treatment for Patients Treated for Choroidal Melanoma From 2002 to 2007

 

Figure 1

 

Survival. Kaplan-Meier analysis showed no difference in all-cause mortality between UCSF and LOOC patients (log rank test, P = 0.503).

Validation of Prognostic Tool

All-cause mortality showed good agreement between predicted and observed survival in both groups of patients (Fig. 2). For the C-index of discrimination, any value of 0.7 or above included within the 95% confidence interval indicates good performance.²⁹ The C-index of discrimination at 5 years after treatment for UCSF patients was 0.72 (95% confidence interval [CI]: 0.61–0.83) and for LOOC patients was 0.80 (95% CI: 0.75–0.85), demonstrating good discrimination performance. The Hosmer-Lemeshow result for all-cause mortality at the 5-year time point for UCSF patients was 0.85 (P = 0.932) and for LOOC patients was 0.86 (P = 0.931), indicating that observed mortality is not significantly different from predicted mortality. 

Figure 2

 

All-cause mortality. All-cause mortality shows good agreement between predicted and observed survival at 5 years of follow-up. Each point on the graph represents a different prognostic group according to survival prediction, that is, with the bottom left circle depicting the group with the worst prognosis, and the top right circle representing the group with the best prognosis. As the survival distribution function tails off exponentially, higher values are more clustered and thus have smaller standard deviation.

As already mentioned, the melanoma-specific mortality was obtained by subtracting patient survival from control survival (Fig. 3). The C-index for UCSF at 5 years was 0.67 (95% CI: 0.54–0.80) and for LOOC was 0.77 (95% CI: 0.71–0.83), indicating good discrimination ability of the model. The Hosmer-Lemeshow result at the 5-year time point for UCSF was 8.83 (P = 0.065) and for LOOC was 7.98 (P = 0.092), indicating no statistically significant difference between predicted and observed survival in both groups of patients (Fig. 3). 

Figure 3

 

Melanoma-specific mortality. For melanoma-specific mortality, there was no statistically significant difference between observed and predicted survival at the 5-year time point.

Discussion

Main Findings

In this study, we externally validated the LUMPO as a tool for predicting survival in patients with choroidal melanoma treated at UCSF. We found that predicted all-cause mortality and metastasis-related mortality correlated well with predictions in all categories of risk at 5 years after treatment. 

Discussion of Findings

While LUMPO predicted metastasis-related mortality well, this was perhaps less robust than all-cause mortality as there was a trend toward significance in the Hosmer-Lemeshow result, although the result did not reach statistical significance. As mentioned, the Hosmer-Lemeshow test is a goodness-of-fit test in which the null hypothesis is that there is no difference between observed and predicted mortality, so a P value greater than 0.05 indicates that observed survival is not significantly different from predicted mortality.^32,33 There are several reasons why metastatic mortality was perhaps predicted less well than all-cause mortality. First, LUMPO was trained on all-cause mortality, not metastatic mortality. Second, the census data used to calculate metastatic mortality were from a short time frame, whereas the study subjects were recruited over a much longer period. Third, LUMPO was trained with patients from the United Kingdom, while the UCSF data were from a different area of the world. Fourth, fewer patients died of metastasis than of unrelated disease so that estimation of the metastatic mortality may have been unduly influenced by a relatively small number of outliers. Finally, the Hosmer-Lemeshow test divides the patients into five categories, some of which had few patients, so the effect of outliers may be exaggerated. 

Comparison of Cohorts

There were differences noted between the UCSF and LOOC cohorts of patients. The UCSF patients were younger, and their tumors were on average thicker but with smaller basal diameter, possibly due to differences in ultrasound technique. For example, at LOOC the ultrasonography was often performed by the attending physician, while at UCSF, the ultrasound scan was typically performed by an ultrasonographer. Survival times were similar in both groups of patients (Fig. 1), and if the tumors had genuinely been thicker at UCSF, one would expect survival to be shorter. This points to likely differences in technique. There were observed differences in extraocular spread, but due to the rarity of extraocular extension, this is unlikely to have influenced the results. The prevalence of ciliary body involvement was the same in both groups of patients, probably because the ciliary body was defined in the same way (i.e., including pars plana) in both groups of patients. Predicted survival may therefore be different in centers where pars plana is not considered to be a part of the ciliary body. At UCSF, patients were treated almost exclusively by proton beam radiotherapy, whereas at LOOC, patients were treated by a variety of other methods, such as plaque radiotherapy, surgical resection, and enucleation. It is beyond the scope of this study to explain the reasons for such differences, but Damato and Heimann³⁴ have explained their treatment selection process elsewhere. It has been shown that treatment type has little effect on survival in these patients.^35,36 The number of tumors with epithelioid cells also differed between the two groups, perhaps due to differences in data acquisition, with specimens mostly obtained by fine-needle aspiration biopsy at UCSF and by enucleation or local resection at LOOC. As enucleation specimens allow for examination of a much larger tumor sample, a greater number of tumors containing epithelioid cells was identified at LOOC, as would be expected. However, despite these observed differences, the LUMPO model performed well for the UCSF cohort. An important aspect of external validation is that the two cohorts are different, and despite these differences the survival predictions were similar. 

Discussion of Methods

The 5-year time period of 2002–2007 was selected to provide sufficient patient numbers, follow-up times, and deaths for statistical analysis. We excluded patients whose tumor did not involve choroid because these are rare, so they were also excluded from the LUMPO models. 

Patients with missing tumor dimensions were excluded because it would not have been possible to determine whether inaccurate survival estimates were the result of incomplete data or failures of LUMPO, so validation of this predictive model may have been compromised. Patients who did not receive treatment were excluded, as prognostication begins at the time of treatment, and tumor characteristics such as dimensions and cell type are not arrested in these patients but may continue to change. 

A 5-year time point was chosen for statistical analysis to allow sufficient follow-up time for all patients within the study group. 

Weaknesses

The main weakness of our study is that many patients treated at UCSF were excluded from the study because of incomplete data. The UCSF data were collected retrospectively, unlike the LOOC data, which were collected prospectively. During the years analyzed at UCSF, all records of tumor dimensions were kept in paper charts, and a large number of these charts could not be located in storage. The missing data may have impaired the analysis, if only by reducing the number of UCSF patients included in the study. Charts from before 2002 were particularly difficult to obtain, which limited the years included in the study. Another weakness is that mortality data were obtained by different methods in the two centers, and this may have skewed the results. Further, it was assumed that longevity in the general populations in the United States and United Kingdom are the same without confirming this statistically. However, reports indicate that life expectancy at birth is similar between the United States and United Kingdom, reported in 2014 as 80.42 years and 79.56 years, respectively,³⁷ and from 1950 to 1955 as 68.58 years and 69.28 years, respectively.³⁸ An unavoidable weakness is that assessment of melanoma cytomorphology is subjective and interobserver repeatability can be low, even within the same center. Finally, there was a lack of genetic tumor data from UCSF. Such data greatly enhance the reliability of survival prognostication, because metastatic disease occurs much more frequently in patients whose tumor shows chromosome 3 loss (with or without gains in 8q) and/or a class 2 GEP, with these aberrations being highly lethal.^17,19,39 It is encouraging to note that even with the lack of genetic data in the UCSF cohort, the LUMPO model performed well. In addition, most of the genetic data used by the present version of LUMPO was obtained by FISH, which has been superseded by more sensitive methods, such as MLPA. Future versions of LUMPO should improve as data from more sensitive genetic tests predominate. However, there is a need for a prognostication tool without the use of genetic information. Patients may not have genetic information available if their tumor is too small to allow biopsy, if prognostic biopsy is not available at their place of treatment, or if they elect not to undergo biopsy. 

Comparison With Other Studies

To our knowledge, LUMPO has not previously been validated externally. The AJCC TNM staging system is increasingly deployed for estimating survival after treatment for choroidal melanoma; however, it uses only anatomic data without taking account of histologic grade of malignancy, genetic tumor type, and life expectancy according to age and sex.^7,40 As for other multivariate analyses of survival, Walter et al. (IOVS 2015;56:ARVO E-Abstract 4334) and Correa and Augsburger (IOVS 2015;56:ARVO E-Abstract 5331) recently reported survival after GEP molecular classification, showing that basal tumor diameter enhances estimation of survival probability. However, these analyses do not seem to have accounted for competing risks, so that the estimates of melanoma-specific mortality are likely to be less accurate than if such bias had been addressed, as has been done with LUMPO. Further, the authors did not perform internal validation with bootstrapping or using a different dataset. For these reasons, their studies should not be used for prognostication as yet. They do, however, confirm the usefulness of multivariate analysis when estimating prognosis.²⁴ These findings are important because it has previously been suggested that GEP alone is sufficient to estimate survival after treatment of choroidal melanoma.³⁹ They lend support to the LUMPO approach, using multiple predictors while taking measures to minimize bias caused by missing data. 

Clinical Implications

Despite observed differences between the UCSF and LOOC cohorts, LUMPO predicted all-cause and melanoma-specific mortality reasonably well. The most important clinical implication of this study is that LUMPO can be used at UCSF, and therefore at other centers using similar methods of tumor evaluation in patients with recorded tumor dimensions that are similar to those of the British population with regard to life expectancy and mortality. Unlike genetic analysis alone, which determines only whether or not a choroidal melanoma is life threatening, LUMPO also gives an approximate indication of when metastatic death is likely to occur should the tumor prove to be fatal.¹⁰ This is important particularly in elderly patients, many of whom die of other causes before metastatic disease has had time to develop. Data provided by LUMPO should not only inform clinical care but may also enhance the selection of patients for any randomized trials evaluating systemic adjuvant therapy. 

Conclusions

It will be important to perform external validation including genetic data once sufficient follow-up time has passed to allow survival analysis. Prognostic biopsies are now routinely being performed and genetic data have been gathered prospectively at UCSF beginning in 2013, and the inclusion of genetic data in future studies should only improve the performance of LUMPO. Upcoming studies could also include a longer follow-up time point, such as 10-year survival, once adequate time has passed. In addition, efforts are in progress to improve LUMPO using a larger dataset and more sensitive methods of genetic analysis. There is, as well, scope for further external validations at different centers to determine whether LUMPO can indeed be used widely. This indicates a need for more standardization in the way tumor size, extent, and grade of malignancy are measured and reported. 

The GEP class 2 group of tumors includes most metastatic deaths; however, it is now known that a small percentage of class 1 tumors also metastasize—the class 1B or “intermediate risk” tumors.²² PRAME has recently been found to confer increased metastatic risk in class 1 or disomy 3 tumors.²³ Mutations such as SF3B1, EIF1AX, and BAP1 also correlate with survival and may therefore have prognostic importance in uveal melanomas with disomy 3 and partial monosomy 3.⁴¹ It is likely that future versions of LUMPO will incorporate these and other new predictors for survival once sufficient data become available. 

Acknowledgments

The authors thank Ann Griffin, PhD, of the University of California-San Francisco Cancer Center Registry for her assistance. 

Disclosure: S.W. DeParis, None; A. Taktak, None; A. Eleuteri, None; W. Enanoria, None; H. Heimann, None; S.E. Coupland, None; B. Damato, None