June 2021
Volume 62, Issue 8
Open Access
ARVO Annual Meeting Abstract  |   June 2021
Impact of Varying Dataset Composition Ratios on the Machine Learning Model Segmentation Performance for Subretinal Hyperreflective Material: A Quantitative and Qualitative Evaluation
Author Affiliations & Notes
  • Hasan Cetin
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Jon Whitney
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Duriye Damla Sevgi
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Jenna Hach
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Sunil Srivastava
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Jamie Reese
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Justis P Ehlers
    Cleveland Clinic Cole Eye Institute, Cleveland, Ohio, United States
  • Footnotes
    Commercial Relationships   Hasan Cetin, None; Jon Whitney, None; Duriye Damla Sevgi, None; Jenna Hach, None; Sunil Srivastava, Allergan (F), Bausch and Lomb (C), Gilead (F), Leica (P), Novartis (C), Regeneron (F), Regeneron (C); Jamie Reese, None; Justis Ehlers, Adverum (C), Aerpio (F), Aerpio (C), Alcon (F), Alcon (C), Allegro (C), Allergan (F), Allergan (C), Boehringer-Ingelheim (F), Genentech (F), Genentech/Roche (C), Leica (C), Leica (P), Novartis (F), Novartis (C), Regeneron (F), Regeneron (C), Santen (C), Stealth (C), Thrombogenics/Oxurion (F), Thrombogenics/Oxurion (C), Zeiss (C)
  • Footnotes
    Support  NIH/NEI K23-EY022947
Investigative Ophthalmology & Visual Science June 2021, Vol.62, 2164. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Hasan Cetin, Jon Whitney, Duriye Damla Sevgi, Jenna Hach, Sunil Srivastava, Jamie Reese, Justis P Ehlers; Impact of Varying Dataset Composition Ratios on the Machine Learning Model Segmentation Performance for Subretinal Hyperreflective Material: A Quantitative and Qualitative Evaluation. Invest. Ophthalmol. Vis. Sci. 2021;62(8):2164.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : Detection of specific features of interest on OCT is strongly linked to training data composition. For many targeted features, large comprehensive ground-truth annotated datasets are not available. Smaller datasets may be more susceptible to class imbalance potentially affecting the machine learning (ML) performance. The purpose of this study was to evaluate the impact of the variable ratios of positive to negative training data on ML performance based on quantitative and qualitative parameters on segmentation of subretinal material (SRM).

Methods : : A U-Net architecture convolutional model was executed and evaluated on training datasets with varying ratios of annotated OCT images containing (positive) and not containing (negative) SRM in neovascular age-related macular degeneration. ML performance based on 5 different ratios of positive (P) and negative (N) data: 30P-70N, 40P-60N, 50P-50N, 60P-40N, 70P-30N was assessed. The quantitative performance evaluation was calculated using F-scores. Qualitative performance evaluation was based on multiple experts’ reviews of the model outputs in a tiled configuration for assessment of optimal segmentation (Figure 1).

Results : The results demonstrated variable model performance related to the training dataset ratio. Based on quantitative model performance, the F-scores ranged from 0.59 to 0.72. The highest performing model based on F-scare was the 70P-30N training set. However, qualitative model performance assessment demonstrated that the 30P-70N (F-score = 0.61) was the preferred training set. In qualitative review, the 70P-30N model demonstrated excellent detection of subretinal material with few false negatives, but with an excess of false positives that was clinically impactful (Figure 1). Conversely, the 30P-70N demonstrated a more conservative segmentation with dramatic reduction in false positives while maintaining minimal false negatives.

Conclusions : This study demonstrates the important of dataset composition and positive/negative sampling ratios in datasets of limited size. In addition, this analysis identifies the potential disconnect between qualitative/practical model performance and quantitative performance metrics.

This is a 2021 ARVO Annual Meeting abstract.

 

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×