June 2022
Volume 63, Issue 7
Open Access
ARVO Annual Meeting Abstract  |   June 2022
Uncertainty estimation of a geographic atrophy OCT segmentation algorithm: How do we identify cases where the algorithm may be mistaken?
Author Affiliations & Notes
  • Luis De Sisternes
    Carl Zeiss Meditec, Inc., Dublin, California, United States
  • Lars Omlor
    Corporate Research and Technology, Carl Zeiss Inc., Dublin, California, United States
  • Warren Lewis
    Bayside Photonics, Inc., Ohio, United States
  • Sophie Kubach
    Carl Zeiss Meditec, Inc., Dublin, California, United States
  • Varsha Pramil
    New England Eye Center, Boston, Massachusetts, United States
  • Harris Asad Sheikh
    New England Eye Center, Boston, Massachusetts, United States
  • Ruikang K Wang
    University of Washington, Seattle, Washington, United States
  • Nadia K Waheed
    New England Eye Center, Boston, Massachusetts, United States
  • Niranchana Manivannan
    Carl Zeiss Meditec, Inc., Dublin, California, United States
  • Footnotes
    Commercial Relationships   Luis De Sisternes Carl Zeiss Meditec Inc., Code E (Employment); Lars Omlor Carl Zeiss Inc., Code E (Employment); Warren Lewis Carl Zeiss Meditec Inc., Code C (Consultant/Contractor), Bayside Photonics Inc., Code E (Employment); Sophie Kubach Carl Zeiss Meditec Inc., Code E (Employment); Varsha Pramil None; Harris Sheikh None; Ruikang Wang Carl Zeiss Meditec Inc., Code C (Consultant/Contractor), Carl Zeiss Meditec Inc., Code F (Financial Support); Nadia Waheed Nidek Medical Products, Boehringer Ingelheim, Topcon, Code C (Consultant/Contractor), Carl Zeiss Meditec Inc., Heidelberg, Nidek Medical Products, Topcon, Code F (Financial Support), Ocudyne, Code I (Personal Financial Interest), Gyroscope Therapeutics, Code S (non-remunerative); Niranchana Manivannan Carl Zeiss Meditec Inc., Code E (Employment)
  • Footnotes
    Support  None
Investigative Ophthalmology & Visual Science June 2022, Vol.63, 3017 – F0287. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Luis De Sisternes, Lars Omlor, Warren Lewis, Sophie Kubach, Varsha Pramil, Harris Asad Sheikh, Ruikang K Wang, Nadia K Waheed, Niranchana Manivannan; Uncertainty estimation of a geographic atrophy OCT segmentation algorithm: How do we identify cases where the algorithm may be mistaken?. Invest. Ophthalmol. Vis. Sci. 2022;63(7):3017 – F0287.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : We developed a geographic atrophy (GA) segmentation model for OCT data based on deep learning. One of the main obstacles for clinical application of this automated algorithm is understanding for which specific cases we can expect reliable results and when it is likely to fail. This work proposes an approach to estimate the regional uncertainty of a deep learning segmentation model.

Methods : We generated uncertainty maps for GA segmentation using a model ensemble trained through cross-validation. As each model in the ensemble was trained with different data, observing the variance in results across them can be used to estimate epistemic uncertainty (unseen manifestations). Data augmentation in the test phase simulating different signal level conditions can estimate the aleatoric uncertainty (signal levels the model is unfamiliar with). Analyzing both sources of uncertainty, we generated maps that relate to model confidence. These maps were evaluated by analyzing the segmentation performance at different levels of uncertainty compared to two expert graders (R1 and R2), excluding from the analysis those regions with uncertainty higher than a chosen level (Figure 1).

Results : We processed 180 OCT scans (PLEX® Elite 9000, ZEISS, Dublin, CA) from GA eyes and 45 from non-GA AMD eyes. Regions of segmentation disagreement between the automated method and each grader had significantly higher values of uncertainty (Figure 2A). Automated segmentation accuracy (Dice Index) increased with lower uncertainty thresholds (excluding larger image regions from the analysis) while the comparison between two graders did not improve (Figure 2B), showing the ability of the uncertainty maps to indicate regions where the algorithm was not confident. Choosing a threshold producing a segmentation performance similar to the intergrader agreement (Dice=0.91) deemed as uncertain approximately 6% of the image on average while maintaining detection sensitivity. Uncertain regions corresponded primarily to locations without GA where the algorithm might make a false positive decision (Figure 2C).

Conclusions : We introduce a method to generate uncertainty maps for an automated GA segmentation model. These maps inform about segmentation confidence and can be used as feedback for manual review.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

Figure 1: Examples of segmentation uncertainty in 3 GA cases

Figure 1: Examples of segmentation uncertainty in 3 GA cases

 

Figure 2: Performance of uncertainty maps

Figure 2: Performance of uncertainty maps

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×