Abstract
Purpose: :
to study the performance effect of retraining a system for automated detection of diabetic retinopathy in digital retinal photographs. We have previously shown that the performance of such a system on 10,000 exams (40,000 photographs) from a true screening population approaches that of retinal specialists (area under the ROC curve(AUC) =0.84)[in press, Diabetes Care].
Methods: :
A new, more recent, previously unseen set of 10,000 consecutive exams (4 retinal photographs, two left and two right) from 6100 unique patients was selected from a diabetic retinopathy screening project in the Netherlands imaged with three types of cameras at ten centers. Inclusion criteria: no previous diagnosis of diabetic retinopathy, no previous visit to ophthalmologist for dilated eye exam, both eyes photographed. One of three retinal specialists evaluated each exam as unacceptable quality, no referable retinopathy, or referable retinopathy. The system selected exams with sufficient image quality, on those, determined presence or absence of referable retinopathy. The algorithm was retrained with the current dataset, and detection performance (AUC) compared to the expert evaluation was evaluated in leave-one-out fashion.
Results: :
human expert classified 253/10,000 exams as referable retinopathy. Total AUC of the system was 0.88. At the optimal threshold for screening, the sensitivity was 94% and the specificity was 59%, the number needing to be screened to miss one case (NSM) was 667. At this point, 7308/10,000 exams had sufficient image quality, 4176/7308 (57%) were true negatives, 11/7308 (0.15%) false negatives, 167/7308 (2.3%) true positives, and 2953/7308 (40%) false positives, and 74/2692 of insufficient quality were positive. Analysis of cases in terms of lesion type is ongoing and will be presented.
Conclusions: :
retraining a supervised system for automated detection of diabetic retinopathy can have a considerable impact on performance. Performance is now close to that of retinal specialists, in this low prevalence dataset, and evaluation on large, validated, available datasets should be vigorously pursued. If systems can be improved even further, they may lead to improved prevention of blindness and visual loss in patients with diabetes, through greater accesibility and cost-effectiveness of screening.
Keywords: diabetic retinopathy • imaging/image analysis: clinical • detection