The ROC regression methodology applied in the current study was originally proposed by Pepe et al.
5 26 and previously used to evaluate the influence of the degree of hearing loss on results of diagnostic tests in audiology, as well as in other applications.
26 27 As this modeling approach has not been previously applied to evaluation of diagnostic tests in ophthalmology, we will describe it in some detail. Further detail can be found in several publications.
5 26 27
The ROC
X,X D (
q) is the probability that a diseased individual with disease-specific covariates
X D and common covariates
X has test results
Y D that are greater than or equal to the
qth quantile of the distribution of tests results from nondiseased individuals. That is, when the specificity of the test is 1 −
q, the sensitivity is ROC
X,X D (
q). An example of disease-specific covariate is severity of the disease, as this covariate is obviously not defined for healthy subjects. In contrast, age is an example of a common covariate, as it is defined for subjects without and those with disease. The effects of
X and
X D can be modeled on ROC
X,X D (
q) by a generalized linear regression model (ROC-GLM model).
27 28 The general ROC regression model can be represented by
\[\mathrm{ROC}_{X,X_{\mathrm{D}}}(q)\ {=}\ g\left({{\sum}_{i{=}1}^{j}}\ {\alpha}_{i}h_{i}(q){+}{\beta}X{+}{\beta}_{\mathrm{D}}X_{\mathrm{D}}\right)\]
The ROC is a function of covariates common to diseased and subjects without disease, covariates specific to diseased subjects, and a function
h(·) which defines the location and shape of the curve. This approach is referred to as parametric distribution-free, as it specifies a parametric model for the ROC curve but does not assume distributions for the test results, which makes it advantageous compared with other modeling procedures.
5 28 The functions
g(·) and
h(·) are chosen so that the ROC curve is monotone, increasing on the unit square. In most applications,
g(·) = Φ, the normal cumulative distribution function,
h 1(
q) = 1 (with coefficient α
1) and
h 2(
q) = Φ
−1(
q) (with coefficient α
2) are generally used, which results in the binormal ROC model
\[\mathrm{ROC}_{X,X_{\mathrm{D}}}(q)\ {=}\ {\Phi}({\alpha}_{1}\ {+}\ {\alpha}_{2}{\Phi}^{{-}1}(q)\ {+}\ {\beta}X\ {+}\ {\beta}_{\mathrm{D}}X_{\mathrm{D}})\]
where the coefficients α
1 and α
2 are the intercept and slope of the ROC curve, respectively. If the coefficient for a specific variable
X(β) is greater than zero, then the discrimination between those with disease and those without increases with increasing values of this covariate. Similarly, if the coefficient for the disease-specific covariate
X D(β
D) is greater than zero, then diseased subjects with larger values of this covariate are more distinct from nondiseased subjects than are diseased subjects with smaller values of
X D.
After the estimation of the parameters using generalized linear models, the area under the ROC curve can be obtained by:
\[\mathrm{AUC}{=}{{\int}_{0}^{1}}\mathrm{ROC}_{X,X_{\mathrm{D}}}(q)\mathrm{dq}\ {=}\ {\Phi}\left(\frac{{\alpha}_{1}\ {+}\ {\beta}X\ {+}\ {\beta}_{\mathrm{D}}X_{\mathrm{D}}}{\sqrt{1\ {+}\ {\alpha}_{2}^{2}}}\right).\]
In the present study, an ROC-GLM model was fitted to assess the influence of the disease-specific covariate severity and the common covariate age on the diagnostic performance of FDT 24-2 and SAP SITA parameters as evaluated by ROC curves. The following ROC regression model was then fitted:
\[\mathrm{ROC}_{X,X_{\mathrm{D}}}(q)\ {=}\ {\Phi}({\alpha}_{1}\ {+}\ {\alpha}_{2}{\Phi}^{{-}1}(q)\ {+}\ {\beta}_{1}\mathrm{FDT}\ {+}\ {\beta}_{2}\mathrm{FDT}\ {\times}\ {\Phi}^{{-}1}(q){+}\ {\beta}_{3}severity\ {+}\ {\beta}_{4}severity\ {\times}\ {\Phi}^{{-}1}(q)\ {+}\ {\beta}_{5}\mathrm{FDT}\ {\times}\ severity{+}\ {\beta}_{6}age\ {+}\ {\beta}_{7}age\ {\times}\ {\Phi}^{{-}1}(q)\ {+}\ {\beta}_{8}age\ {\times}\ \mathrm{FDT}),\]
where FDT is a binary variable indicating the type of test (SAP SITA was used as the reference test),
severity is the variable indicating severity of glaucomatous damage as measured by percentage loss of rim area, and
age is a variable indicating patient’s age. Interaction terms between the variables and Φ
−1(
q) were included to allow the effects of the covariates to differ by varying amounts depending on the
FPRq (or specificity 1 −
q), that is, to influence the shape of the curve. Interaction terms between FDT and severity and between FDT and age were included to assess whether the influence of disease severity and age was similar or different between FDT 24-2 and SAP SITA tests.
Parameters were estimated using probit regression. To obtain confidence intervals for regression parameters, a bootstrap resampling procedure was used (
n = 500 resamples).
29 As measurements from both eyes of the same subject are likely to correlate, the use of standard statistical methods for parameter estimation can lead to underestimation of standard errors and to confidence intervals that are too narrow.
30 Therefore, to account for the fact that both eyes of some subjects were used for analyses, the cluster of data for the study subject was considered as the unit of resampling when calculating standard errors. This procedure has been used in other studies to adjust for the presence of multiple correlated measurements from the same unit.
27 29
Statistical analyses were performed on computer (Stata ver. 9.0; StataCorp., College Station, TX; and SPSS ver. 13.0; SPSS Inc., Chicago, IL). The α level (type I error) was set at 0.05.