Abstract
Purpose :
Artificial intelligence (AI) may be able to aid in ophthalmic disease detection, including cost-effective screening for glaucoma. Previously, an AI algorithm [1] was trained to detect glaucoma in optical coherence tomography (OCT) retinal nerve fiber layer (RNFL) probability maps with a score of 0 (no glaucomatous optic neuropathy (GON) or visual field (VF) loss) to 100 (definite GON/VF loss). In this study, we designed a GUI (graphical user interface) to assess the impact of this additional AI information on physician accuracy, confidence, speed, and sensitivity/specificity of glaucoma diagnosis.
Methods :
The GUI was designed using PsychoPy, an open-source, Python software, and was deployed via Pavlovia, an online platform for behavioral experiments. Six experts (glaucoma faculty/fellows in ophthalmology) were shown 24 randomized instances of 2 stimuli (12 each): either 1) just a patient’s OCT report (‘No AI Present’, Fig 1) or 2) a patient's OCT report, a heatmap showing regions used by the AI to make its prediction, and the AI’s score (‘AI Present’, Fig 2). Experts were then prompted to give their own numerical diagnostic score (0-100), the next probable step in their management plan (discharge, monitor, treat), and a rating on their response confidence (0 to 10, 10 being most confident). The reference standard (ground truth) was based on prior expert review of clinical and imaging data. To compute results, scores > 50 were glaucoma; scores < 50 were not glaucoma.
Results :
Trials with ‘AI Present’ had an average accuracy of 91.7%, average confidence of 8.71, average response time of 13.7 seconds, average sensitivity of 0.95, and average specificity of 0.91. Trials with ‘No AI Present’ had an average accuracy of 83.3%, average confidence of 8.57, average response time of 13.9 seconds, average sensitivity of 0.74, and average specificity of 0.94. The increase in average sensitivity with ‘AI Present’ was statistically significant (Mann-Whitney U Test, p = 0.02).
Conclusions :
The results suggest that participants’ average accuracy, confidence, speed, and sensitivity improved with ‘AI Present’; sensitivity improved by a statistically significant amount. This retrospective study indicates that introducing AI information along with the OCT report can aid in disease detection. Future work will focus on improving the AI’s impact on specificity, since this a major barrier to effective glaucoma screening paradigms. [1] Thakoor et al., IEEE TBME 2021.
This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.