A hierarchical optical coherence tomography annotation workflow with crowds and medical experts

Abstract

Purpose : There is a heavy burden to annotate clinical images due to the cost and availability of medical experts. To address this problem, we proposed a hierarchical annotation workflow in which medical experts review aggregated crowdsourced annotations, using dense annotation of optical coherence tomography (OCT) images from age-related macular degeneration (AMD) patients as an example.

Methods : Annotation was performed on a cloud-based platform (Labelbox) which distributed images to remote crowds and medical experts. OCT B-scans with ≥ 9 averages from 20x20° volume scans of AMD patients were randomly selected. Two medical experts annotated 25 representative B-scans with rich pathology. In a training session, 27 labellers read through an annotation guideline and practiced on 15 of the B-scans. B-scans with color-coded agreements and disagreements were presented to the crowd, visualizing their discrepancy to the expert annotations. Performance, compared to the medical experts’ annotations, was measured quantitatively by a weighted and adaptively normalized mean intersection over union (IOU) for each of the annotated structures (WANI). The top 10 performers on the remaining 10 OCT B-scans (test session, Fig. 1. Left) were selected for larger batches of annotation work. Additional 897 B-scans were labeled by 5 labellers each. Simultaneous truth and performance level estimation (STAPLE) was applied to aggregate the 5 copies of crowd labels. Medical experts qualitatively reviewed the STAPLE result to identify common mistakes to further train the crowd. The crowd and experts annotated another 8 representative B-scans in a follow-up analysis session (Fig. 1. Right).

Results : STAPLE results show above average IOU in individual anatomic structures, while outperforming all crowd labellers in WANI, though lower than experts. On average, it took experts 13.5 minutes to annotate one B-scan in contrast to 23.8 seconds to review one STAPLE result. Improvements in STAPLE WANI were observed in the follow-up session, as the crowd gained experience and further trained on the feedback from experts.

Conclusions : The proposed hierarchical annotation workflow with crowds and medical experts could reduce the burden on medical experts in extensive clinical annotation tasks.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

IOUs for individual anatomic features and WANI score in test and follow-up sessions