June 2023
Volume 64, Issue 8
Open Access
ARVO Annual Meeting Abstract  |   June 2023
A novel tool for generating gold standard to validate image segmentation
Author Affiliations & Notes
  • Anse Vellappally
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Guido Gerig
    Department of Computer Science & Engineering, NYU Tandon School of Engineering, Brooklyn, New York, United States
    Department of Biomedical Engineering, NYU Tandon School of Engineering, Brooklyn, New York, United States
  • Palaiologos Alexopoulos
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Sarah Segal
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Ronald Zambrano
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Allison Toyos
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Shijie Li
    Department of Computer Science & Engineering, NYU Tandon School of Engineering, Brooklyn, New York, United States
  • TingFang Lee
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
    Division of Biostatistics, Department of Population Health, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Jiyuan Hu
    Division of Biostatistics, Department of Population Health, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
  • Joel S Schuman
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
    Department of Biomedical Engineering, NYU Tandon School of Engineering, Brooklyn, New York, United States
  • Gadi Wollstein
    Department of Ophthalmology, NYU Langone Health, NYU Grossman School of Medicine, New York, New York, United States
    Department of Biomedical Engineering, NYU Tandon School of Engineering, Brooklyn, New York, United States
  • Footnotes
    Commercial Relationships   Anse Vellappally None; Guido Gerig None; Palaiologos Alexopoulos None; Sarah Segal None; Ronald Zambrano None; Allison Toyos None; Shijie Li None; TingFang Lee None; Jiyuan Hu None; Joel Schuman Zeiss, Code P (Patent); Gadi Wollstein None
  • Footnotes
    Support  NIH R01-EY030770, P30EY013079, An unrestricted grant from Research to Prevent Blindness
Investigative Ophthalmology & Visual Science June 2023, Vol.64, OD60. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Anse Vellappally, Guido Gerig, Palaiologos Alexopoulos, Sarah Segal, Ronald Zambrano, Allison Toyos, Shijie Li, TingFang Lee, Jiyuan Hu, Joel S Schuman, Gadi Wollstein; A novel tool for generating gold standard to validate image segmentation. Invest. Ophthalmol. Vis. Sci. 2023;64(8):OD60.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : Determining the true segmentation of complex structure such as the lamina cribrosa (LC) from OCT scans is challenging due to the intervening structure of the beams and pores, and the shadow casted upon by the blood vessels. In this study we are proposing the use of Simultaneous Truth and Performance Level Estimation (STAPLE) method, which is based on an expectation-maximization algorithm and has been originally implemented in CT and MRI scans, to generate a best estimated ground truth for validating LC segmentation.

Methods : Twenty spectral-domain OCT (Leica, Chicago, IL) scans of varying quality acquired in-vivo from 19 healthy non-human primates were used for the study. A single 2D slice of the LC was extracted from each scan and was assigned to five trained raters to perform manual segmentation in a masked fashion. Five sets of manual segmentations for each OCT scan along with the original scan were fed as input to the STAPLE software to generate a probabilistic map and consensus segmentation. STAPLE also provides an estimate of the performance level of each rater when compared to the probabilistic estimate reported as specificity and sensitivity. The quantitative analysis of pores from both the STAPLE method and the majority agreement of raters (≥3 out of 5) and all raters (5 of 5) were obtained using Fiji and compared using paired t-test.

Results : All raters showed high specificity due to the relatively large areas of the black background (Figure, Table 1). However, large variation was shown in the sensitivity ranging between 0.787 and 0.331 with the lowest sensitivity demonstrating poor performance in detecting the laminar pores. Using the STAPLE method, a significantly higher number of pores were detected in comparison with both methods of raters’ agreement with a subsequent higher area of pores. (Table 2). However, for pore shape descriptive parameters like aspect ratio and roundness, no significant difference was detected between the methods.

Conclusions : The STAPLE method enables generating a more comprehensive gold standard accounting for the varying performance of the raters. The ability of STAPLE for performance level estimation can be used to identify raters who need additional training.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

 

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×