Real-Time Feature-Guided Image Fusion of Posterior-Segment Intrasurgical Optical Coherence Tomography and Digital Surgical Microscopy

Abstract

Purpose : Intraoperative real-time volumetric OCT (4D-OCT) is a cutting-edge imaging modality for improved performance in ophthalmic surgery. Currently, 4D-OCT is integrated with traditional color surgical microscopy side-by-side, requiring the surgeon to pay a significant cognitive overhead to integrate the presented information. As a solution, a real-time pipeline for the fusion of 3D structural data and 2D color data is conceived, producing a single live visualization (Fig. 1) as a more effective presentation of the visual cues surgical guidance.

Methods : Real-time fused rendering of the recorded retina is conducted on custom-built software based on the Vortex library running on an Intel® Core™ i9-9900K CPU @ 3.60GHz and a NVIDIA GeForce RTX 2060 GPU. The fusion pipeline (summarized in Fig. 1) begins with intensity-based segmentation of features in the OCT data (Fig. 1, surface detection), including the retinal surface+surgical tool (yellow) and retinal pigmented epithelium (RPE, green). Rendering of a fused visualization is achieved via raycasting into these surfaces. A specular shader is applied to the tool and retinal surfaces, while color shading is applied to the RPE and tool. Additionally, the RPE shader is modulated by the neighboring OCT intensity, adding structural OCT features. Computation time was measured as the per-volume computation time for a 256x700x450 volume.

Results : Figure 2 demonstrates fusion of detail from microscopy imaging (blood vessels, tool contrast, pigmentation) with that from 4D-OCT data (tool position, surface topology). Importantly, the fusion of these details has not compromised their visibility. Visualization pipeline performance was measured as 43.93ms per volume (22.8 fps).

Conclusions : Intraoperative 4D-OCT imaging and stereo color microscopy can be fused to produce a single novel visualization combining relevant cues from each in an effective manner for retinal visualization for surgery. Critically, this process can be executed at real-time frame rates, a requirement for any technique intended to enhance live image guidance.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Visualization of the image fusion process for a soft-tip tool on approach to a depression in the retinal surface.

 

Time series example of the fusion of 2-D color microscopy (row 1) and 3D-OCT (row 2) into a single comprehensive visualization (row 3) for the approach of a soft-tip tool to a retinal depression.