A Hybrid Deep Learning and Optimal 3D Graph Search Approach Improves Retinal Layer Segmentation in Very Thin Retina

Abstract

Purpose : Automated OCT layer segmentation is essential for evaluating eyes with optic nerve or retinal injury. Current methods are unreliable when the retinal layers become thin. The errors may not be apparent and can lead to incorrect conclusions of the severity of disease and change over time. We hypothesized a 3D hybrid approach that combines nnU-Net and graph-search optimization could achieve robust OCT segmentation of the ganglion cell inner plexiform layer (GCIPL) in eyes with significant retinal thinning due to non-arteritic anterior ischemic optic neuropathy (NAION) and glaucoma.

Methods : We used 40 OCT macular scans from 20 NAION participants (20 affected and 20 unaffected fellow eyes) for training the hybrid approach (Deep LOGISMOS, Fig. 1). The nnU-Net module was trained to represent retinal layer spatial patterns as probability maps serving as inputs to the subsequent 3D graph-search optimization module, which considers retinal layer anatomy as a constraint to provide more accurate layer surfaces to achieve ground truth. Reference manual tracings were created (for training and evaluation) by modifying the incorrectly segmented areas in the OCT images using our previously reported gradient-based graph-search algorithm.

Results : Independent 14 NAION and 21 glaucoma OCT macular scans were tested on the trained segmentation hybrid approach and compared to the manual ground-truth segmented GCIPL. Our hybrid approach yielded surfaces to the ground-truth with root-mean-square errors (RMSE; relative deviation from manually traced surfaces) of 4.65 µm (top GCIPL surface) and 4.59 µm (bottom GCIPL surface). The improvement was significant over the previous gradient-based graph-search segmentation that yielded top/bottom GCIPL surface RMSE of 7.16 µm and 7.05 µm, respectively (p < 0.05; Fig. 2).

Conclusions : A new hybrid method combining deep learning and 3D graph optimization was able to segment GCIPL more accurately and will help provide more correct diagnosis and assessment of disease progression.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Flow chart of the hybrid method: obtain the region of interest, create 3D region-based cost images, and apply graph-based optimization.

 

(A) Surface root-mean-square errors for the top and bottom GCIPL surface of the hybrid method vs. the 3D graph-search method alone, referenced to the ground-truth manual tracing. (B) An example of a B-scan overlaid with segmentation in the three methods.