Purchase this article with an account.
Yun Ling, Richard A. Bilonick, Hiroshi Ishikawa, Gadi Wollstein, Joel S. Schuman; Ordinal Measurement Error Model for Assessing Agreement Among Raters for Glaucoma Progression. Invest. Ophthalmol. Vis. Sci. 2012;53(14):6375.
Download citation file:
© ARVO (1962-2015); The Authors (2016-present)
An ordinal measurement error model (OMEM) that describes agreement in terms of a slope (polychoric correlation) and intercept (threshold) was used to describe the agreement among 5 raters for glaucoma progression. This is an improvement over using kappa because, in general, agreement cannot be described as a single parameter.
The agreement among 5 raters (A, B, C, D, E) judging progression in eyes of 60 subjects was modeled using a common factor measurement error model for ordinal data. It assumes that subjects had an unobserved (latent) progression factor on a continuous scale. Each rater had a latent continuous scale for judging but only ordinal outcomes (no progression, progression) were observed. The model for agreement includes a scale (slope, -1 < β < 1) that also measures the polychoric correlation between the common factor and each observed rating and simultaneously describes the precision. Thresholds (intercepts, α) convert the unobserved rater's continuous judgment to ordinal outcomes. Since there are two rating categories, one threshold is needed per rater. The probability of making a judgment of progression increases with the amount of true progression μ:P(rating = progression) = P(z > α - β*μ)where the probabilities come from a unit standard normal distribution. R software with OpenMx was used to describe the structural equation model (SEM) for the OMEM (Figure 1) and estimate the parameters.
Full information maximum likelihood (FIML) estimates of the most important and relevant functions of the model parameters are shown in Table 1. Rater E has the steepest slope, indicating E is the fastest in transition from non-progressor to progressor. Also E happens to have the lowest intercept, indicating E tends to rate more subjects as progressors. The estimated correlation between both eyes was ρ = 0.29, 95% CI (-0.06, 0.64). Subjects A, B, and C were very similar in their ratings and different from D and E.
SEM accounts for the correlations between both eyes of each subject, and, when FIML is used, missing ratings. Correctly describing agreement requires two parameters: a scale that describes the rate of transition and thresholds that control when the transitions occur.
This PDF is available to Subscribers Only