An assessment of subjective meibography image grading between observers and the impact formal gland interpretation training on inter-observer agreement of grading scores.

Abstract

Purpose : Many characteristics of meibomian glands can be graded subjectively by the observer including atrophy and segmentation. Past studies have shown subjective measures of meibography imaging to be variable between observers. This study looked at inter-observer agreement for atrophy and a novel scale of meibomian gland segmentation. How formal training of meibography may improve inter-observer agreement was investigated.

Methods : Twenty meibography images (10 upper eyelid images, 10 lower eyelid images) from three commercial meibographers were selected from multiple clinics and graded by 9 skilled dry eye clinician observers. Meibography images were graded on a whole integer ordinal scale (0 to 4) using the Pult Atrophy scale and the LEO Segmentation scale. The observers were then asked to complete a meibography interpretation training and regrade the same set of meibography images. Fleiss' kappa test was performed to see how each grader compared to a key that was agreed upon by 3 experienced graders.

Results : When assessing gland atrophy, mean agreement was 0.34 (0.23-0.51) before training and 0.34 (0.05-0.55) after training. Five graders improved in agreement with a change in k of between 0.04 to 0.11, two graders’ agreement decreased by -0.01 to -0.08, one observer decreased by -0.22 only because that observer did not grade all of the images and any missed grades were scored as disagreement. One observer did not complete the second round of grading. Mean segmentation agreement was 0.065 (-0.01-0.21) before training and 0.13 (0.05-0.23) after training. Agreement improved for three observers from 0.02 to 0.11 and decreased for four observers by -0.01 to -0.21.

Conclusions : Meibography images are currently graded subjectively bringing into question the reliability between observers. Image quality is paramount to interpretation. Intense training on meibography interpretation improves grading scores for gland segmentation and gland atrophy. Atrophy continues to be the most reliable measure between observers. However, with continued training and use of new grading scales, clinicians may grade meibography images more reliably. Future objective and artificial intelligence/machine learning quantification of meibomian gland morphology would obviate the need for subjective grading.

This is a 2020 ARVO Annual Meeting abstract.

 

 

LEO Segmentation Scale