Abstract
Purpose :
A reliable reference standard is critical for training and evaluation of deep learning models. This is especially relevant in glaucoma, as guidelines for diagnosis are not well-defined. This study investigates the extent to which agreement on individual optic nerve head (ONH) features is associated with agreement on overall glaucoma risk assessment.
Methods :
A dataset of 508 color fundus photos (508 patients) was graded by a rotating panel of 3 glaucoma specialists (out of 10). Images were assessed for 12 glaucomatous ONH features (see Table 1) and the overall glaucoma risk. Images were first independently graded by each specialist. In case of disagreement, the same 3 specialists reviewed the image a second time, this time with access to the annotations and comments from the other 2 specialists. Each image was reviewed up to a maximum of 6 times.To investigate the relative importance of individual ONH features in the consensus formation process for overall glaucoma risk assessment, we used a logistic regression model. For each individual review (N=1123), we determined whether agreement was reached for each of the 12 features. The resulting 12 binary agreement states in a given round were treated as independent predictor variables. Agreement on the overall glaucoma risk assessment for that round was used as the dependent outcome variable.
Results :
For 5 out of 12 features, panel agreement was a significant positive predictor for panel agreement on overall glaucoma risk assessment in the same round, listed in decreasing order by relative importance based on Odds Ratio (OR): notch (p = 0.006, OR = 2.157), retinal nerve fiber layer (RNFL) defect (p < 0.001, OR = 1.678), baring (p < 0.001, OR = 1.622), vertical cup-to-disc (p < 0.001, OR = 1.621), and laminar dot sign (p = 0.003, OR = 1.557).
Conclusions :
We identified 5 features for which agreement among panel members is positively associated with panel agreement on overall glaucoma risk assessment. Our results contribute towards a more nuanced understanding of the relative importance of individual evaluation criteria for consensus-based glaucoma risk assessment.
This abstract was presented at the 2019 ARVO Annual Meeting, held in Vancouver, Canada, April 28 - May 2, 2019.