Methods for Manual Segmentation of Hyper-resonant Foci to Identify a Ground Truth for Deep Learning Models

Abstract

Purpose : The presence of hyper-reflective foci (HRFs) in the outer retina has been associated with age-related macular degeneration (AMD) and diabetic retinopathy (DR). We have previously employed a deep-learning (DL) model to perform semantic segmentation of HRFs. Here, we conducted a retrospective, observational clinical study to identify a ground truth for the DL model using manual segmentation.

Methods : Optical Coherence Tomography (OCT) images were retrieved of patients who received care at the retina service at the University of Bonn. The dataset included 3644 OCT (3044 AMD & 600 DR) images. Two separate teams of graders reviewed the images; team A consisted of one ophthalmologist, and team B consisted of a medical student and ophthalmologist. ImageJ was used to manually annotate the boundaries of HRFs found in the outer nuclear and/or outer plexiform layers for each OCT image. We measured the variability of manual segmentation of HRFs between two sets of graders and evaluated the accuracy of the DL model compared to manual segmentation by both teams.

Results : Overall, fewer HRFs were marked by the team B compared to team A; in fact, there were images where no HRF was annotated by the team B. Out of 3644 images, 3557 images (97.6%) had annotations which matched across both teams. When excluding images with zero overlap, the average intersection over union (IoU) was 0.285, and the average precision (AP) was 0.343 measured over 2303 images. When the DL model was trained with annotations by team A as the ground-truth, the IoU was 0.33, and the AP was 0.75. When the DL model was trained with the annotations by team B as the ground-truth, the IoU was 0.53, and the AP was 0.78.

Conclusions : There was considerable variability in the manual annotation of HRFs by the two teams of graders. This may be due to a low signal-to-noise ratio in several OCT images and challenges in delineating the outer retina from adjacent structures. A higher IoU was noted in the second set of annotations due to conservative delineation of HRFs and a larger team size. Therefore, a DL approach is essential for accurate and precise segmentation of biomarkers such as HRFs in OCT images.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

OCT image of a patient with AMD with HRFs located in the outer retinal layers

 

Manual annotations of HRFs based on OCT image in Figure 1: blue (Team A), red (Team B), and green (annotations common to both teams)