Clinical validation of a machine-learned algorithm for detection of diabetic retinopathy (DR) and diabetic macular edema (DME) in fundus images

Abstract

Purpose : To measure the performance of a machine-learned algorithm (MLA) in classifying DR and detecting DME in 45° fundus images.

Methods : A single 45° fundus image centered between the macula and optic disc was retrospectively collected for 850 diabetic individuals from EyePACS. Images were selected to obtain a prespecified distribution of DR and DME based on grades from EyePACS teleretinal service.

Images were prospectively reviewed by three graders, each with experience grading DR/DME in more than 10,000 images. Graders used the ICDR Disease Severity Scale, assigning a DR grade from 1 of 6 options (No, Mild, Moderate, Severe, Proliferative, Ungradable) and a DME grade from 1 of 3 options (No, Yes, Ungradable). Adjudication was performed by the same graders in case of any disagreement. The distribution of adjudicated grades was [23.5% No, 23.7% Mild, 23.7% Moderate, 13.9% Severe, 15.2% Proliferative, 0% Ungradable] for DR and [69.9% No, 28.3% Yes, 1.8% Ungradable] for DME.

To evaluate the performance of the MLA, its grades for DR and DME were compared to the adjudicated grades.

Results : The MLA classified 13 of 850 (1.5%) images as ungradable for either DME or DR. Of these, 12 were DME ungradable but DR gradable; 1 was ungradable for both. The graders classified 15 images as DME ungradable (6 overlapped with the MLA) and no images DR ungradable. The MLA and the graders were within one level of agreement for DR in 818 of 849 (96.3%) cases gradable for DR.

In each of the following metrics, ungradable classification was considered as referrable. The MLA:
● Detected referrable DR (i.e. Moderate, Severe, or Proliferative DR) with a sensitivity of 90.4% (95% CI: 87.3-93.0%) and a specificity of 97.5% (95% CI: 95.5-98.8%).
● Classified 1 of 130 (0.8%, CI: 0.02-4.21%) cases as less than Moderate DR, which graders classified as Proliferative DR.
● Classified 19 of 325 (5.8%, CI: 3.56-8.98%) cases as No DME and less than Moderate DR, which graders classified as vision-threatening DR.

Conclusions : The MLA performed well in identifying referrable DR cases and had a low ungradability rate. This strongly supports its potential usefulness in a clinical screening setting. The MLA’s ability to identify images with the most severe levels of DR and DME suggest it could be used effectively to screen for cases needing urgent referral.

This abstract was presented at the 2019 ARVO Annual Meeting, held in Vancouver, Canada, April 28 - May 2, 2019.