Real-time tracked intraoperative OCT using synthetically trained machine learning models

Abstract

Purpose : Intraoperative optical coherence tomography (iOCT) visualization of ophthalmic surgeries is limited by the narrow field of view (FOV) of OCT relative to that of the surgical microscope, requiring precise manual repositioning by a trained operator. Approaches have been developed for automatic ophthalmic instrument tracking; however, these require markers on surgical instruments, use specialized imaging systems, or are designed for post-op analysis. We created a computer vision system that utilizes the surgical microscope video feed to track ophthalmic surgical instruments in real time via an automatically annotated machine learning dataset.

Methods : Our system uses a machine learning model trained on a synthetic image dataset we generated by 3D rendering models of surgical instruments (Fig. 1). This pipeline allows us to generate many distinct training images and to automatically annotate features. We selected cataract surgery as an example procedure with high velocity tool movement relative to surgical field size. The model was trained on 20000 synthetic images (640x640) of 4 tool classes: phacoemulsification (phaco) probes, knives, forceps, and needles. The trained model was integrated into a clinical iOCT system: the model output was used to steer the OCT galvo scanners and acquire OCT volumes at the tool location.

Results : In testing with a phantom eye and real cataract tools (Fig. 2), the system correctly tracked the tool for 89.5% of the 8384 frames in which a tool was visible. Over the course of 115 tracking sessions (3.6 hours) with phantom and porcine eyes, the model had a processing time per frame median (interquartile range) of 15 (12-21) ms, corresponding to a tracking rate of 67 (48-83) Hz, which exceeds the volume acquisition rate of the OCT system.

Conclusions : We demonstrate the successful implementation of a real-time surgical instrument tracking system via a machine learning model trained on synthetic microscope views. Our synthetic data approach allows us to rapidly update our training set and provides another level of flexibility for future automated tool tracking.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.

 

Overview of the synthetic data pipeline and tracking system. The model outputs the tracked microscope stream with an iris (blue box), a primary tool (pink box), and OCT target coordinates (orange cross).

 

Microscope and OCT images showing a successful tracking session with a model eye phantom and all 4 classes of tool.