Evaluation of automated software grading of diabetic retinopathy and comparison with manual image grading - an accuracy and cost effectiveness study

Abstract

Purpose: Current diabetic retinopathy screening is labor intensive and involves manual grading of digitized retinal images by trained graders. Commercially available automated grading softwares may reduce this workload, but their accuracy compared to manual grading has never been examined using large scale, real world data. We therefore sought to determine if automated grading software detects diabetic retinopathy as accurately as manual graders, the current gold standard. We will compare 4 commercially available programs - iGrading (Medalytix), Retmarker (Critical-Health), EyeCheck (IDx) and EyeMark (EyeNuk). All investigators are independent of commercial software development and have no commercial interest or intellectual property in automated diabetic retinopathy screening.

Methods: The automated software packages will grade 18,000 real world screening episodes, which have already been manually graded, from patients attending diabetic retinopathy screening programmes in a hospital in London, UK (Homerton).

Results: As of 30th November 2013, 18,000 patient episodes have been downloaded from the Homerton diabetic retinopathy screening servers and will be batch processed on dedicated servers in mid-December. A pilot study of 1,340 patient screening episodes revealed 100% sensitivity for proliferative disease and 91% for diabetic macular oedema comparing one automated software with manual grading as the reference standard. Detailed results will be available by January and we expect these to be similar to the pilot results above. Cost effectiveness will be evaluated based on the expected cost per true positive case of retinopathy and cost per Quality Adjusted Life Year (QALY).

Conclusions: Automated grading could assist with screening diabetes patients by identifying low and high risk groups, thereby concentrating limited resources on high risk groups with more severe retinopathy. For example, an algorithm combining both automated screening flagging ‘disease’ cases which could then be confirmed by human graders and referred onwards if necessary could reduce much repetitive human input. This project will provide detailed estimates of the accuracy of 4 commercially available automated grading softwares in detecting diabetic retinopathy, as compared to manual graders.

Keywords: 499 diabetic retinopathy • 550 imaging/image analysis: clinical • 496 detection