Local feature matching transformers for intraoperative OCT en face and fundus image registration

Abstract

Purpose : High-speed intrasurgical OCT systems generate enormous amounts of data. To better leverage this data, for example, by averaging or continuous field-of-view expansion, robust and fast registration methods are required. Therefore, we present a registration technique based on Local Feature Matching with Transformers (LoFTR) for robust and precise registration of OCT en face patches and fundus images at video rate of inference times

Methods : The core of our method is the use of LoFTR for displacement estimation. First, the dimensionality of volumetric OCT data is reduced by generating en face feature maps (refer Fig. 1A). Then, the homography matrix is asserted (refer Fig. 1B). The pipeline was trained on eight 4D-miOCT data sets, each with 20 to 100 volumes. Translation and rotation errors were assessed on patches from an independent widefield fundus image. Furthermore, we measured the inferential time for integrating each patch into the mosaic.

Results : Results are presented in a mosaic pattern, illustrating the large stitched en face image at various stitching iterations (refer Fig. 2).
To evaluate lateral position errors, we measured the Euclidean distance from the ground truth position. The median lateral position error was under 25 pixels, and for rotational errors, the median was less than 10 degrees. Errors are calculated relative to the top-left corner of the large reference fundus image. Inference times, without additional optimization, for each sub-patch, including merging into the mosaic, was less than 50 ms.

Conclusions : Our method skillfully addresses the need for rapid processing, particularly in the context of the high-speed requirements imposed by state-of-the-art 4D-miOCT systems. In practice, although the median lateral registration error was 25 pixels and 10 degrees, our approach is well suited for registering small groups of volumes against each other, rendering the observed errors insignificant and enabling fast and accurate lateral registration. Potential extensions have the potential to improve the accuracy and performance of the pipeline for real-time volumetric applications, such as axial corrections at the A-scan level.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.

 

Lateral registration pipeline: (A) en face feature map extraction, and (B) local feature matching network, akin to LoFTR.

 

Compilation of 74 en face images: A corresponds to Step 10, B to Step 30, C to Step 50, and D to Step 74.