June 2022
Volume 63, Issue 7
Open Access
ARVO Annual Meeting Abstract  |   June 2022
Real-Time Framewise Semantic Understanding of Instruments and Tissues Via Deep Learning in Vitreoretinal Surgery
Author Affiliations & Notes
  • Rogerio Garcia Nespolo
    Department of Ophthalmology and Visual Sciences - Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois, United States
    Richard and Loan Hill Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, Illinois, United States
  • Darvin Yi
    Department of Ophthalmology and Visual Sciences - Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois, United States
    Richard and Loan Hill Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, Illinois, United States
  • Yannek Leiderman
    Department of Ophthalmology and Visual Sciences - Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois, United States
    Richard and Loan Hill Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, Illinois, United States
  • Footnotes
    Commercial Relationships   Rogerio Garcia Nespolo Microsurgical Guidance Solutions LLC, Code I (Personal Financial Interest), USSN: 63/183424 (provisional patent application), Code P (Patent), WO/2020/163845, Code P (Patent); Darvin Yi None; Yannek Leiderman Alcon, Code C (Consultant/Contractor), Genentech, Code C (Consultant/Contractor), Allergan, Code C (Consultant/Contractor), Regeneron, Code C (Consultant/Contractor), RegenXBio, Code C (Consultant/Contractor), Microsurgical Guidance Solutions LLC, Code I (Personal Financial Interest), USSN: 63/183424 (provisional patent application), Code P (Patent), WO/2020/163845, Code P (Patent)
  • Footnotes
    Support  Chancellor's Translational Research Initiative (CTRI) - University of Illinois Chicago
Investigative Ophthalmology & Visual Science June 2022, Vol.63, 2058 – F0047. doi:
  • Views
  • Share
  • Tools
    • Alerts
      ×
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      Rogerio Garcia Nespolo, Darvin Yi, Yannek Leiderman; Real-Time Framewise Semantic Understanding of Instruments and Tissues Via Deep Learning in Vitreoretinal Surgery. Invest. Ophthalmol. Vis. Sci. 2022;63(7):2058 – F0047.

      Download citation file:


      © ARVO (1962-2015); The Authors (2016-present)

      ×
  • Supplements
Abstract

Purpose : Real-time imaging in vitreoretinal surgery directly impacts the handling of surgical instrumentation and the decision-making process of the surgeon. This experimental study tested the hypothesis that deep learning neural networks can concurrently track and segment instruments and target tissue landmarks within the retina when attached to a surgical microscope.

Methods : A hundred and one (101) vitreoretinal procedures consisting of core vitrectomy, membranectomy, and endolaser application were employed to train and validate an instance segmentation convolutional neural network. Three vitreoretinal surgeons manually annotated the following features from six-hundred and six (606) fundus frames: vitrector, forceps and endolaser tooltips, optic disc, fovea, retinal tear and detachment, fibrovascular proliferation, endolaser spot, the area where endolaser was applied, and macular hole. Five-fold cross-validation was employed to assess the performance of the model.

Results : The model detected and classified the vitrector tooltip with a mean area under the precision-recall curve (AUPR) of 0.972±0.009. Segmentation of target tissues such as optic disc, fovea, and macular hole reached mean AUPR values of 0.928±0.013, 0.844±0.039, and 0.916±0.021 respectively (Table 1 and Figure 1). The post-processed image was rendered at a resolution of 1920x1080 pixels at 38.77±1.52 frames per second when attached to an intraoperative visualization system.

Conclusions : Deep learning neural networks can detect, classify, and segment tissues and instruments during different phases of vitrectomy surgery in real time. We propose a model that creates a framework to develop surgical guidance tools that may guide surgical decision-making in real time. Potential applications include (1) warning of unintended instrument-tissue interactions and potential intraoperative complications, (2) data extraction for the control of equipment parameters such as the vitrector cutting rate, and (3) retroactive data acquisition of instrument maneuvers for surgeon’s skills analysis.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

Table 1 - Mean AUPR for the detection and segmentation of each feature

Table 1 - Mean AUPR for the detection and segmentation of each feature

 

Figure 1 – Post-processed images from our model. Top left: Retinal detachment segmented and vitrector tooltip location detected. Top right: Detection and segmentation of features during endolaser photocoagulation. Bottom: Detection and segmentation of features during membranectomy.

Figure 1 – Post-processed images from our model. Top left: Retinal detachment segmented and vitrector tooltip location detected. Top right: Detection and segmentation of features during endolaser photocoagulation. Bottom: Detection and segmentation of features during membranectomy.

×
×

This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.

×