Reliability of physicians’ measurements when manually annotating images of microbial keratitis

Abstract

Purpose : To determine the reliability of physicians to measure the size of microbial keratitis on slit-lamp images.

Methods : Prospectively-enrolled participants with fungal and bacterial keratitis underwent slit-lamp imaging at their first clinical encounter at two academic eye hospitals (Aravind Eye Care Systems, AECS, and University of Michigan, UM). Two physicians outlined the borders of the stromal infiltrate on diffuse, white-light images using ImageJ software (National Institute of Health, Bethesda, MD). Because white-to-white distance was not measured, image pixels were used as the unit of analysis. The physicians were masked to participants’ clinical information. Physicians’ annotations were evaluated for reliability using DICE similarity coefficient (DSC) and intraclass correlation coefficient (ICC). The DSC is the proportion of twice the number of pixels identified by both graders divided by the sum of pixels identified in each image. (Figure 1) The DSC ranges from 0 (no overlap) to 1 (perfect overlap) and a coefficient >0.8 is considered very good.

Results : A total of 101 images from 101 participants were studied. Participants were 39.6% female with an average age of 49.6±13.6 years. Organisms causing keratitis were fungal in 64 cases (63%), bacterial in 14 cases (14%), both bacterial and fungal in 1 case (1%), acanthamoeba in 1 case (1%), and unidentified (culture yielded no growth) in 21 cases (21%) of the sample. The area of the stromal infiltrate traced by the first physician grader was on average 454,679 pixels (standard deviation, SD=521,627; median=276,962) and on average 548,343 pixels (SD=553,313; median=350,472) by the second physician grader. DSC was on average 0.81 (SD=0.20; median: 0.88). The ICC was 0.94 (95% CI, 0.86-0.97) showing good physician grader agreement.

Conclusions : Physicians can reliably identify and trace the infiltrate for microbial keratitis from slit-lamp imaging; however, physicians do have some differences when manually tracing images. Training and review of test images with graders will be important if manual tracing is to be used to design automated image analysis software.

This abstract was presented at the 2019 ARVO Annual Meeting, held in Vancouver, Canada, April 28 - May 2, 2019.

 

Figure 1: Corneas with infectious ulcer. Comparison of manual tracing of reader 1 (red area) and reader 2 (blue area); left: DSC=0.98; right: DSC=0.70