Real-World Assessment and Clinician Acceptance of AI in GA Measurement

Abstract

Purpose : This project explores the application of Artificial Intelligence (AI) models in the measurement of Geographical Atrophy (GA) to monitor its progression. Despite successful development, there is a significant translational gap between the creation of AI models and real-world deployment. Our study aimed to evaluate two main areas: the real-world performance of an AI model in measuring GA and the level of clinician agreement in using AI-generated reports for decision-making.

Methods : The AI model, trained on AREDS2 Fundus Autofluorescence (FAF) images annotated at the Wisconsin Reading Center, showed a dice coefficient of 0.93 in a clinical trial test dataset. We then deployed this model at Moorfield’s Eye Hospital, London to analyze their retrospective clinical dataset. Clinicians annotated FAF images of their GA patients to create a ground truth, which was compared with AI's predictions. In a second independent project, we investigated clinician comfort with AI-generated reports in a prospective clinical trial at University of Wisconsin, Madison. The AI reports were used to verify if participants met specific area-based inclusion criteria, thus determining their eligibility for enrollment. The AI report with GA segmentation mask and measurements were reviewed by ophthalmologists to confirm accuracy. Inaccurate segmentation masks were escalated to reading center experts for confirmation of GA area.

Results : The retrospective dataset included 158 FAF images. The difference in GA area between AI and clinician annotation was 0.48 (0.31, 0.64 95% CI) mm², with a mean dice coefficient of 0.90 (n = 48 ). The prospective clinical trial dataset included 17 FAF images. Ophthalmologists agreed with the AI measurement in 12 (70.5%). Reasons for disagreement included when Ophthalmologists found the AI misinterpreted foveal involvement and AI missing small areas of GA which were corrected by RC graders.

Conclusions : The AI model maintained its performance in a real-world clinical setting, as evidenced by clinician annotations, demonstrating its robustness. Ophthalmologists showed a high level of agreement in relying on AI reports for decision-making. Feedback received will be used to enhance future reports. While its effectiveness in aiding GA treatment is yet to be established, the model's potential in tracking longitudinal progression presents a promising avenue for future research.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.