Abstract
Purpose :
Advances in artificial intelligence (AI) models for video have enabled the automatic calculation of granular performance metrics for cataract surgeries. However, the correlation between AI metrics and cataract surgical skill is unclear. This study investigates the ability of AI approaches to distinguish surgical skill performance between novice and expert cataract surgeons and correlates these AI-generated surgical performance metrics to those generated by expert human judges.
Methods :
28 resident and 29 attending routine cataract surgical videos were anonymously collected. For each video, six AI-generated metrics were calculated: phacoemulsification probe decentration, eye decentration, zoom level change, and instrument-specific total path length, maximum velocity, and area covered. Expert human raters independently rated the videos using Objective Structured Assessment of Cataract Surgical Skill (OSACSS), which has 20 sub-items on a 5-point scale, with higher scores indicating greater surgical skill. Statistical differences between machine & human-rated scores between attending and trainee surgeons were tested by t-tests, and the correlations between them were examined by Pearson correlation coefficients.
Results :
Phacoemulsification probe & irrigation/aspiration probe had significantly lower total path lengths, maximum velocities, and area metrics in attending videos compared to trainee videos. Attending surgeons exhibited significantly better phacoemulsification centration and eye fixation. Most AI metrics correlated with human-rated OSACSS scores, including tool-specific metrics (total path length, max velocity, area) and metrics related to microscope control (eye decentration: -0.394). AI-generated metrics with corresponding OSACSS subitems also exhibited significant negative correlations (eye decentration: -0.65, phacoemulsification probe centration: -0.64, Figure).
Conclusions :
Automatically generated AI-metrics effectively distinguish between attending and trainee surgeries and correlate with human expert evaluations of surgical performance. These metrics, generated rapidly from cataract surgical videos, can provide performance feedback for trainees, holding the potential for creating an automated, objective feedback system in ophthalmology training, ensuring standardized and consistent analysis of surgical techniques.
This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.