A novel tool for generating gold standard to validate image segmentation

Abstract

Purpose : Determining the true segmentation of complex structure such as the lamina cribrosa (LC) from OCT scans is challenging due to the intervening structure of the beams and pores, and the shadow casted upon by the blood vessels. In this study we are proposing the use of Simultaneous Truth and Performance Level Estimation (STAPLE) method, which is based on an expectation-maximization algorithm and has been originally implemented in CT and MRI scans, to generate a best estimated ground truth for validating LC segmentation.

Methods : Twenty spectral-domain OCT (Leica, Chicago, IL) scans of varying quality acquired in-vivo from 19 healthy non-human primates were used for the study. A single 2D slice of the LC was extracted from each scan and was assigned to five trained raters to perform manual segmentation in a masked fashion. Five sets of manual segmentations for each OCT scan along with the original scan were fed as input to the STAPLE software to generate a probabilistic map and consensus segmentation. STAPLE also provides an estimate of the performance level of each rater when compared to the probabilistic estimate reported as specificity and sensitivity. The quantitative analysis of pores from both the STAPLE method and the majority agreement of raters (≥3 out of 5) and all raters (5 of 5) were obtained using Fiji and compared using paired t-test.

Results : All raters showed high specificity due to the relatively large areas of the black background (Figure, Table 1). However, large variation was shown in the sensitivity ranging between 0.787 and 0.331 with the lowest sensitivity demonstrating poor performance in detecting the laminar pores. Using the STAPLE method, a significantly higher number of pores were detected in comparison with both methods of raters’ agreement with a subsequent higher area of pores. (Table 2). However, for pore shape descriptive parameters like aspect ratio and roundness, no significant difference was detected between the methods.

Conclusions : The STAPLE method enables generating a more comprehensive gold standard accounting for the varying performance of the raters. The ability of STAPLE for performance level estimation can be used to identify raters who need additional training.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.