July 2018
Volume 59, Issue 9
Open Access
ARVO Annual Meeting Abstract  |   July 2018
Artificial intelligence in retinopathy of prematurity: clinical validation of a fully automated deep learning system (i-ROP DL) for plus disease diagnosis
Author Affiliations & Notes
  • J. Peter Campbell
    Casey Eye Institute, Oregon Health & Science University, Portland, Oregon, United States
  • James M Brown
    Massachusetts General Hospital, Boston, Massachusetts, United States
  • Susan Ostmo
    Casey Eye Institute, Oregon Health & Science University, Portland, Oregon, United States
  • R.V. Paul Chan
    University of Illinois, Chicago, Chicago, Illinois, United States
  • Jennifer Dy
    Northeastern University, Boston, Massachusetts, United States
  • Deniz Erdogmus
    Northeastern University, Boston, Massachusetts, United States
  • Stratis Ioannidis
    Northeastern University, Boston, Massachusetts, United States
  • Jayashree Kalpathy-Cramer
    Massachusetts General Hospital, Boston, Massachusetts, United States
  • Michael F Chiang
    Casey Eye Institute, Oregon Health & Science University, Portland, Oregon, United States
  • Footnotes
    Commercial Relationships   J. Peter Campbell, None; James Brown, None; Susan Ostmo, None; R.V. Chan, Alcon (C), Allergan (C), Bausch & Lomb (C), Visunex (C); Jennifer Dy, None; Deniz Erdogmus, None; Stratis Ioannidis, None; Jayashree Kalpathy-Cramer, INFOTECH Soft Inc (C); Michael Chiang, Clarity (S), Novartis (C)
  • Footnotes
    Support  R01 EY019474 from the National Institutes of Health, Bethesda, Maryland; 2. P30EY010572 from the National Institutes of Health, Bethesda, Maryland; 3. NSF SCH-1622679; 4. Unrestricted funding from the Research to Prevent Blindness
Investigative Ophthalmology & Visual Science July 2018, Vol.59, 3936. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      J. Peter Campbell, James M Brown, Susan Ostmo, R.V. Paul Chan, Jennifer Dy, Deniz Erdogmus, Stratis Ioannidis, Jayashree Kalpathy-Cramer, Michael F Chiang; Artificial intelligence in retinopathy of prematurity: clinical validation of a fully automated deep learning system (i-ROP DL) for plus disease diagnosis. Invest. Ophthalmol. Vis. Sci. 2018;59(9):3936.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : We compare the performance of i-ROP DL, a fully automated deep learning (DL) system for plus disease diagnosis, with the performance of expert ROP clinicians.

Methods : Using a deep convolutional neural network (Deep-ROP) described elsewhere (Brown et al, ARVO 2018), as part of the ongoing “Imaging and Informatics in ROP” (i-ROP) study, we developed a fully automated open source deep learning system (i-ROP DL) for plus disease diagnosis. We compared the performance of i-ROP DL on an independent test set of 100 images that were previously graded, and ranked in order of disease severity, by 8 international ROP experts using methods previously published. The diagnostic performance of the DL algorithm was compared to experts using weighted kappa statistics. A continuous score was created (from 1 to 9) using the DL output and compared to the expert ordered ranking of disease severity.

Results : Figure 1 shows the weighted kappa statistics for each of the 8 graders, the RSD, the consensus diagnosis, and the i-ROP DL diagnosis. The weighted kappa score for i-ROP DL compared to the RSD was 0.92, better than 6 of the 8 experts. In the test set, i-ROP DL accurately diagnosed 91/100 (91%) images correctly, whereas 8 experts had an average accuracy of 82% (range 77%-94%, previously published). Figure 2 displays the i-ROP DL derived severity score for each of the 100 images compared to order of disease severity, as determined by experts using pairwise comparisons.

Conclusions : The i-ROP DL system classified plus disease as well as international ROP experts. Incorporation of this technology into routine ROP care could provide an objective method of documenting and monitoring disease severity in ROP.

This is an abstract that was submitted for the 2018 ARVO Annual Meeting, held in Honolulu, Hawaii, April 29 - May 3, 2018.

 

Figure 1. Weighted kappa statistics between 8 individual expert readers, the consensus of 8 readers, the reference standard diagnosis (RSD), and the i-ROP DL program.

Figure 1. Weighted kappa statistics between 8 individual expert readers, the consensus of 8 readers, the reference standard diagnosis (RSD), and the i-ROP DL program.

 

Figure 2. i-ROP score as a function of disease severity among 100 ranked images in the test set.

Figure 2. i-ROP score as a function of disease severity among 100 ranked images in the test set.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×