Abstract
Purpose:
Traditional analyses of classifier performance used to compare clinical test methods such as specificity, sensitivity or ROC curves are often inadequate because the sharp classification boundary does not reflect the smooth gradient of certainty in a zone between possible classifier outcomes. In this study we develop a novel approach to analysing test performance by including a clinically determined loss function to allow the boundary between test outcomes to become fuzzy.
Methods:
An aggregate loss function (a monotonic sigmoidal curve), the “loss function”, was developed by the authors in conjunction with other electrophysiologists for the significance of the Arden Ratio (AR) measurement of the EOG. 54 eyes of 27 patients were analysed with an automated method of EOG calculation previously described and compared to results from traditional manual calculation. In brief, an approximation to a square wave was automatically fitted to saccadic deflections by nonlinear least squares (nls) and robust regression (rr) techniques. The ARs (ratio of the maximum amplitude in the light divided by the minimum amplitude in the dark) determined from both methods were compared to manually measured results. The automated methods were compared to the manual method by traditional classifier measures and our Uncertainty Classifier.
Results:
At an AR cutoff of 200%, sensitivities were 89% nls and 96% rr, while specificities were 92% nls and 92% rr. These differences were not statistically significant. The area under the ROC curve (AUC) was 0.957 nls and 0.982 rr. This difference was significant at the p=0.091 level using DeLong’s test. Correlation between the ARs determined by Pearson’s correlation coefficient was 0.945 (nls vs manual) and 0.948 (robust vs manual). This was not statistically significant. Using our uncertainty classifier, the correlation between the diagnostic confidence level was 0.927 (nls vs manual) and 0.943 (rr vs manual) which was significant at the p=0.08 level using Zou’s technique.
Conclusions:
The Uncertainty classifier encapsulates the clinical weight applied to a test result especially where there is a quantifiable amount of uncertainty. This statistical technique has utility in comparing tests and may find a role in transforming inputs before they are used in machine learning systems.
Keywords: 510 electroretinography: non-clinical •
468 clinical research methodology •
459 clinical (human) or epidemiologic studies: biostatistics/epidemiology methodology