Abstract
Purpose :
The use of a laser indirect ophthalmoscope for retinal laser therapy is challenging as magnifying lenses produce enlarged and inverted images. To be safe and successful, this procedure requires prolonged training and confidence. Herein, we propose an augmented reality (AR)-assisted system to provide guidance by identifying the retinopathy (e.g., diabetic retinopathy, retinal tears) through deep learning and overlaying a hologram of retinal landmarks and areas to be treated.
Methods :
We developed an AR app using the Microsoft HoloLens 2 that overlays the hologram of retinal landmarks on the magnifying lens through feature matching (Figure 1). We used a scale-invariant feature transformation from OpenCV to detect the feature points on the magnified images of the retina captured by HoloLens 2. These feature points were matched to the corresponding color fundus retinal image (CFP) using a fast library for the approximate nearest neighbor matching and filtered out for good matching points to eliminate false matches. The feature matching provides the location and scale of the magnified retinal region to HoloLens 2 for accurate positioning and scaling of the hologram to be overlaid on the magnifying lens. In addition to the visualization-based guidance from the hologram, we integrated contextual guidance for alerting clinicians on the regions that need to be treated and avoided with the laser by visualizing the textual information in the hologram (Figure 2).
Results :
Using 160x120 resolution of the image frame, our edge processing achieved an average of 73.5±3.48 feature points (4.71±0.22% of all feature points) for matching, and the latency resulted in an average of 156±6.71ms per image frame.
Conclusions :
Our AR-assisted guidance system can provide visualization support to clinicians and trainees by accurately overlaying the hologram of magnified retinal landmarks and alerting surgeons to the regions that need to be avoided or treated to preserve vision. We plan to conduct user study experiments to evaluate the effectiveness and feasibility of our system in clinical settings.
This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.