July 2018
Volume 59, Issue 9
Open Access
ARVO Annual Meeting Abstract  |   July 2018
Assisted reads for diabetic retinopathy using a deep learning algorithm and integrated gradient explanation
Author Affiliations & Notes
  • Rory Sayres
    Google, Mountain View, California, United States
  • Ankur Taly
    Google, Mountain View, California, United States
  • Ehsan Rahimy
    Google, Mountain View, California, United States
    Palo Alto Medical Foundation, San Carlos, California, United States
  • Katy Blumer
    Google, Mountain View, California, United States
  • David Coz
    Google, Mountain View, California, United States
  • Naama Hammel
    Google, Mountain View, California, United States
  • Jonathan Krause
    Google, Mountain View, California, United States
  • Arunachalam Narayanaswamy
    Google, Mountain View, California, United States
  • Zahra Rastegar
    Google, Mountain View, California, United States
  • Derek Wu
    Google, Mountain View, California, United States
  • Shawn Xu
    Verily, Mountain View, California, United States
  • Lily Peng
    Google, Mountain View, California, United States
  • Dale Webster
    Google, Mountain View, California, United States
  • Footnotes
    Commercial Relationships   Rory Sayres, Google (E); Ankur Taly, Google (E); Ehsan Rahimy, Google (C); Katy Blumer, Google (E); David Coz, Google (E); Naama Hammel, Google (C); Jonathan Krause, Google (E); Arunachalam Narayanaswamy, Google (E); Zahra Rastegar, Google (C); Derek Wu, Google (E); Shawn Xu, Verily (E); Lily Peng, Google (E); Dale Webster, Google (E)
  • Footnotes
    Support  None
Investigative Ophthalmology & Visual Science July 2018, Vol.59, 1227. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Rory Sayres, Ankur Taly, Ehsan Rahimy, Katy Blumer, David Coz, Naama Hammel, Jonathan Krause, Arunachalam Narayanaswamy, Zahra Rastegar, Derek Wu, Shawn Xu, Lily Peng, Dale Webster; Assisted reads for diabetic retinopathy using a deep learning algorithm and integrated gradient explanation. Invest. Ophthalmol. Vis. Sci. 2018;59(9):1227.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : Recent machine learning methods have produced models which can grade retinal fundus images for diabetic retinopathy (DR) with doctor-level accuracy. The impact of these models on DR diagnosis in assisted-read settings has not yet been measured. We investigated whether surfacing model predictions and explanatory saliency maps ("masks") to doctors improved DR grading accuracy, speed, and confidence.

Methods : We recruited 9 ophthalmologists to read 1,806 cases each for DR severity. Readers graded 45° fundus images centered around the macula. The image sample was representative of the diabetic population, and was adjudicated by 3 retina specialists (1 also a reader) for ground truth grades.

Doctors read each image in one of 3 conditions: Unassisted, Grades Only, or Grades+Masks. The Grades Only condition surfaced a histogram of scores from a deep learning model trained to detect DR. The Grades+Masks condition also showed a mask generated using the integrated gradients explanation method, which indicated pixels contributing to the highest-scoring DR grade. Experimental conditions were counterbalanced across readers.

Results : Readers graded DR more accurately with model assistance than without (p < 0.01, logistic regressions). Accuracy lift was driven by cases with some degree of DR. Doctors also reported significantly higher confidence in their grade with assistance. Read times increased overall with assistance.

Both Grades-Only and Grades+Masks conditions showed a similar pattern of improvements over Unassisted reads. For most cases, Grades Only was as effective as Grades+Masks. Masks were inapplicable in the no DR cases. Masks provided additional benefit over grades alone in cases with: some DR and low model certainty; low image quality; and PDR with features that were possible to miss (e.g. PRP scars).

Model assistance also shifted the operating points of doctors: They became much more sensitive to true positives, and either neutral or slightly less specific.

Conclusions : Deep learning models can improve the accuracy of, and confidence in, DR diagnosis in an assisted read setting. Explanation masks can further improve diagnosis in some, but not all, cases.

This is an abstract that was submitted for the 2018 ARVO Annual Meeting, held in Honolulu, Hawaii, April 29 - May 3, 2018.

 

Experimental conditions. Left: Unassisted; center: Grades Only; right: Grades+Masks. Graders could toggle assistance on/off.

Experimental conditions. Left: Unassisted; center: Grades Only; right: Grades+Masks. Graders could toggle assistance on/off.

 

Summary metrics for DR grading accuracy, confidence, and time on task, ± 95% confidence intervals.

Summary metrics for DR grading accuracy, confidence, and time on task, ± 95% confidence intervals.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×