Annotation of Surgical Tool Depth in Vitreoretinal Surgical Videos: Agreement and Performance Between Vitreoretinal Surgeons vs. Non-Surgeon Graders

Abstract

Purpose : AI implementation in vitreoretinal surgery may enhance surgical training, provide safety mechanisms, and enable objective research from surgical videos. However, there is a lack of standards for the appropriate surgical annotator and annotation methods. This study examines the variability among surgical video graders of varying clinical experience by assessing the agreement and performance between vitreoretinal surgeons and non-surgeons in the annotation of surgical tool depth.

Methods : Three vitreoretinal surgeons and three lay scientists independently annotated an identical set of surgical videos comprising 66,920 frames. To annotate tool depth, each grader categorized the distance of the surgical tool tip from the retina into four zones: far, intermediate, near, and contact. Frames were arranged chronologically, with video playback. Computer Vision Annotation Tool (CVAT) was used for recording annotations. A deep learning model was trained using frames where the surgeons agreed (57,050). Performance was evaluated by comparing percent accuracy against the predictions of the trained model.

Results : Non-surgeons annotated a significantly higher percentage of frames than surgeon graders. However, surgeons showcased greater agreement than any two non-surgeons (57,050 vs. 30,134 frame consensus). Surgeons and non-surgeons agreed on only 9,020 frames. Compared with the trained model for instrument depth prediction, no grader achieved accuracy greater than 60%. Surgeons demonstrated more consistency and slightly higher accuracy in their performance compared to non-surgeons.

Conclusions : Our data suggests that non-surgeon annotators may benefit from additional training for more accurate and uniform annotations in evaluating tool depth. Graders performed sub optimally against the trained model, suggesting the existing method of using CVAT for surgical tool depth annotation in surgical videos is insufficient. There is a need for custom annotation software and workflow modification for refined annotation strategies.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.

 

Figure: Metrics for graders performing depth annotation task and their performance compared to a trained AI model