Accuracy of Automated Retinal Software for Diabetic Retinopathy Detection in Type I Diabetics vs Human Graders

Abstract

Purpose : The incidence of diabetic retinopathy (DR) is increasing, placing stress on healthcare systems. We evaluate an automated diabetic retinopathy detection software against two certified DR graders for referable Type I Diabetes Mellitus (T1DM) DR.

Methods : Non-mydriatic images were captured at an angle of 45° and a flash setting of 100 watt second using a Canon CR2 Plus AF (Tokyo, Japan) 21 MegaPixel retinal camera. A total of 236 subjects (472 eyes, 39.91% male, avg 11.48 years with DM) were analyzed using the EyeArt™ (Eyenuk, Inc., Los Angeles, CA) automated DR screening software. EyeArt™ reports DR severity as no referral (NR), referable disease (RD), and vision threatening disease (VTD). Two certified graders (CG) assessed referable and/or vision-threatening disease using the 5-level International Clinical Diabetic Retinopathy Scale. Intergrader agreement was evaluated with Cohen’s kappa and differences between CGs were arbitrated by a third grader. Duration of analysis was measured for CGs and the AI. Statistical measurements were computed only if both graders and EyeArt™ determined images as gradable. Subjects were further stratified by age (≤ 25 and >25 y/o) to evaluate the effects of retinal sheen in younger subjects on reading accuracy.

Results : We analyzed 446 eyes from 223 subjects. 13 subjects/26 eyes (avg 24.45 years with DM) were deemed ungradable by the graders and/or software. Intergrader agreement for subject referral was κ = 0.829 (95% CI 0.596-1.062, p<.0005). The software achieved 85.71% sensitivity (95% CI 42.13-99.64) and 92.59% specificity (95% CI 88.25-95.71) in detecting referable DR by patient. Of 16 false positives, 14 subjects had ≥1 dot hemorrhage noted by one or both graders but were not considered referable disease. Grading time was significantly faster for the software (27.39s AI vs. 88.39s manual, p<.00001).

Conclusions : Specificity was greater in subjects ≤ 25 than subjects >25. Given the number of false positives, evaluation must include differentiation of hemorrhage type (dot, flame, IRMA), incorporation of the Early Treatment Diabetic Retinopathy Study Grid to assess the geographical location of DR lesions, and differentiation of DR status beyond NR, RD, and VTD to improve the accuracy of DR referral.

This is a 2020 ARVO Annual Meeting abstract.

 

Comparison of sensitivity, specificity, PPV, NPV, and accuracy between subjects ≤ 25 and > 25 y/o.

 

ROC curves comparing AI analysis by patient age.