A Multi-vendor Dataset and Standardized Evaluation Framework for Retinal Cyst Segmentation

Abstract

Purpose: Optical coherence tomography (OCT) has become the most frequently used imaging method for retinal diseases. In clinical practice physicians evaluate pathologic OCT morphologies, e.g. cysts, as a basis for diagnostic decisions. However, precise monitoring of disease progression and treatment success is difficult due to a lack of validated automated segmentation algorithms for these 3D OCT morphologies. We present a multi-vendor ground truth dataset for training and testing of automated cyst segmentation algorithms to be made publically available as part of a segmentation grand challenge.

Methods: The presented dataset contains 16 OCT scans from 2 major scanners (Cirrus, Spectralis), divided into 3 sets. The first set contains 8 scans (4+4) for algorithm training, annotated manually by 2 expert graders. The other 2 sets were annotated by 1 grader: 4 scans for algorithm testing and another 4 for future challenge testing. Annotations were performed in the B-scan plane using a proprietary system and stored as separate 3D ground truth data.<br /> Two accuracy measures are defined for objective comparison of algorithm results to ground truth. One examines the overlap between system results and ground truth based on the Sørensen-Dice index. The second, based on Hausdorf distance, examines the distance between the system cyst boundaries and ground truth. In addition algorithm performance is computed based on 2 criteria, clinical significance and size of cyst.

Results: Manual annotations for ground truth of the training set by 2 graders presented:<br /> A mean number of annotated cysts per scan of 147 and 549, with a mean grader difference of 8±11 and 22±32 for Spectralis & Cirrus scans respectively. Further analysis showed that disparity between cyst annotations only occurred in cases of high number of small cysts per scan.

Conclusions: First manual annotations for ground truth training data for cyst detection algorithms gave us similar results for both graders. The difference in cyst numbers between vendors directly reflects the lower number of B-scans in the Spectralis protocol. Annotations for the next 2 vendors are being completed and will double our database. The ability to objectively rank the currently expected 15 different detection algorithms shall allow a better understanding of positive and negative aspects of each method and improve performance of retinal disease progression and treatment success monitoring.