Abstract
Purpose: :
To test and validate a measure of consensus among low vision rehabilitation (LVR) service providers in their estimation of the rehabilitation potential (LVRP) of individual patients with varying clinical presentations of vision impairment, co-morbidities, and functional deficits.
Methods: :
Ten diverse paper cases were reviewed online by 50 LVR service providers who were asked to estimate the probability of successful rehabilitation outcomes using a 5-category ordinal rating scale. LVRP was modeled as an unbounded continuous latent variable that is normally distributed across service providers for each case. The mean of the distribution represents the expected LVRP estimated for the case by the population of LVR service providers and the standard deviation represents a measure of consensus within the population for that case. Means and standard deviations were estimated for each case from ROC analyses of all possible pairings of cases. The cumulative frequency of using at least each rating category across all providers was transformed to a z-score for each case. When transformed to z-scores, the scatter plot of values for a pair of cases (i.e., z-transform of the ROC curve) is expected to be a line if the distributions are normal. The slope of the line is equal to the ratio of the standard deviations of the two distributions and the intercept is proportional to the difference between the means. The mean and standard deviation of the first case were arbitrarily set to 0 and 1, respectively, and the means and standard deviations for the other cases were estimated relative to case 1.
Results: :
The scatter plots of z-scores for all pairs of cases were well fit by lines, confirming the assumption of normal distributions. The distribution of estimated standard deviations ranged from 0.57 for the case with the highest level of consensus to 1.0 for the case with the lowest level of consensus. The median standard deviation for the 10 cases was 0.71. The means of the cases ranged from -1.74 to 0.14 (in case 1 standard deviation units). The median case had a mean LVRP of -0.35.
Conclusions: :
ROC analyses of cumulative distributions of LVRP ratings concur with the assumption of normally distributed estimates across LVR service providers of each patient’s LVRP. Consequently, the estimation of means and standard deviations from z-transformed ROC curves is valid and cases can be positioned on an interval scale for LVRP and the estimated standard deviation is a measure of consensus among raters.
Keywords: low vision • clinical research methodology