Assessing Usability and Clinical Utility of GLANCE, an Artificial Intelligence-based Tool for Predicting Glaucoma Visual Field Progression

Abstract

Purpose : Artificial intelligence (AI) has the potential to improve risk stratification in glaucoma, but interface design and usability of AI-based clinical decision support tools have been understudied. GLANCE is a visualization tool depicting predicted mean deviation of visual fields (VFs) based on a deep learning model trained on optical coherence tomography (OCT) images, with the potential to reduce VF testing frequency (van den Brandt A, et al., Eurographics Workshop on Visual Computing for Biology and Medicine, 2020). Here, we conducted a pilot study of clinicians using an updated prototype of GLANCE to assess usability, trust in the tool, and utility for clinical management.

Methods : Our updated prototype of GLANCE included dynamic visualization of the AI model output (predicted VF mean deviation) alongside prior VFs and model predictions, intraocular pressure measurements, and clinical history (Fig.1). An online survey consisting of six patient cases with the GLANCE tool was disseminated to clinicians at the University of California San Diego. Participants were asked to comment on whether GLANCE provided additional useful information and whether it impacted their clinical management using 5-point Likert scale items (Fig. 2). They also completed a System Usability Scale (SUS) questionnaire, a standardized instrument for assessing usability.

Results : Seven clinicians (two attendings, two fellows, two residents, and one optometrist) participated in the pilot. The mean SUS score of GLANCE was 62.5/100, which is normalized into the 35th percentile. The mean SUS score among respondents ages 35-44 (75) was greater than ages 65-74 (45). The mean score among female respondents (67.5) was greater than male respondents (57.5). Respondents generally agreed that they trusted GLANCE and that it provided useful information, but disagreed that they would decrease VF testing frequency (Fig.2).

Conclusions : Clinicians agreed that GLANCE provided useful information but were generally not yet willing to allow AI tools to significantly alter their clinical management. Further research is needed to understand how to increase trust in AI models, improve the usability of visualization tools and interfaces, and optimize the integration of AI-based tools into ophthalmic workflows among a diverse group of clinicians.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

GLANCE tool for an example case.

 

Clinician perceptions of GLANCE.