June 2020
Volume 61, Issue 7
Open Access
ARVO Annual Meeting Abstract  |   June 2020
Challenges in evaluating clinical deployments of Deep Learning Assisted Diagnostics for Diabetic Retinopathy Screening
Author Affiliations & Notes
  • Gregory Wolff
    Work done at Google Health via Advanced Clinical, Mountain View, California, United States
  • Rory Sayres
    Google, California, United States
  • Varun Gulshan
    Google, California, United States
  • Kasumi Widner
    Google, California, United States
  • Jonathan Krause
    Google, California, United States
  • Dushyant Jadeja
    Google, California, United States
  • Kim Ramasamy
    Aravind Eye Hospital, Madurai, India
  • Parida Haemoglobin
    Aravind Eye Hospital, Madurai, India
  • Footnotes
    Commercial Relationships   Gregory Wolff, Google Inc (E); Rory Sayres, Google Inc (E); Varun Gulshan, Google Inc (E); Kasumi Widner, Google Inc (E); Jonathan Krause, Google Inc (E); Dushyant Jadeja, Google Inc (E); Kim Ramasamy, Aravind Eye Hospital (E); Parida Haemoglobin, Aravind Eye Hospital (E)
  • Footnotes
    Support  None
Investigative Ophthalmology & Visual Science June 2020, Vol.61, 2045. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Gregory Wolff, Rory Sayres, Varun Gulshan, Kasumi Widner, Jonathan Krause, Dushyant Jadeja, Kim Ramasamy, Parida Haemoglobin; Challenges in evaluating clinical deployments of Deep Learning Assisted Diagnostics for Diabetic Retinopathy Screening. Invest. Ophthalmol. Vis. Sci. 2020;61(7):2045.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : Previous research has demonstrated that the clinical effectiveness of computer aided diagnostic (CAD) systems may differ significantly from the results obtained through carefully controlled research trials. Our goal was to conduct a case study of a deep learning (DL) based CAD system deployed in a clinical setting, to understand the real world factors that may affect the performance, use, and evaluation of these tools in practice.

Methods : A prospective single-reader study with n=1000 patients, using a DL system designed to assist in the diagnosis of Diabetic Retinopathy (DR). For each patient, three non-mydriatic fundus images of each eye were taken and sent electronically to an experienced reader (clinician) for diagnosis. After initial grading (five-point DR scale), the reader viewed the diagnosis provided by the DL system (five-point scale) and was asked to provide their final diagnosis. At any point the reader was allowed to consult a retina specialist.

Results : After seeing the DL results, the reader changed their grade in about 3% of cases, resulting in a 20% increase in the number of Proliferative Diabetic Retinopathy cases detected. The reader consulted a specialist on 4% of cases before and an additional 8% after seeing the DL output. The reader’s final diagnosis from the DL diagnosis in 20% of the cases. A separate group of retina specialists adjudicated these discordant cases. The false positive rate for the reader was almost unchanged at <5% before and after seeing the DL output despite a high false positive rate of 25% for the DL system. Note that the DL output was conservative in adopting the most severe grading for any one of the 3 images per eye regardless of whether the system judged that image “gradable”. DL false positives decreased from 20% to 10% if the “ungradable” images (38% of the total) are removed from analysis.

Conclusions : Pragmatic factors such as whether and when a system rejects an image as being ungradable or the availability of an immediate expert consult can have major impacts on the clinical usefulness and efficacy of DL CAD systems in practice. Pilot deployments and initial field tests of these systems should monitor the effect of these factors over time in order to create the safest systems with the most value to patients.

This is a 2020 ARVO Annual Meeting abstract.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×