Explaining Deep Learning Models for Low Vision Prognosis

Abstract

Purpose : Low vision rehabilitation improves quality-of-life for visually impaired patients, but referral rates fall short of national guidelines. Automatically identifying patients with poor visual prognosis from electronic health records (EHR) could allow targeted referrals to low vision services. The purpose of this study was to build, understand, and explain deep learning artificial intelligence models that predict visual prognosis using EHR data.

Methods : We identified 5547 patients with low vision (defined as best documented visual acuity (VA) <20/40) from EHR from 2009-2018, with ≥ 1 year of follow-up. From the EHR, we extracted ophthalmology free-text notes and structured data, such as demographics, billing/procedure codes, medications, and eye exam findings. To predict whether low vision patients would still have low vision a year later, we developed deep learning models that used 1) structured inputs, 2) free-text progress notes represented by standardized clinical concepts extracted using named entity recognition (NER), and 3) a combination of the two. Performance metrics including area under the receiver operating curve (AUROC) and F1 score were evaluated on a held-out test set. We then evaluated models using Local Interpretable Model-Agnostic Explanations to determine which input features were most important to the predictions.

Results : Among the 5547 patients, 40.7% (N=2258) never improved to better than 20/40 over one year of follow-up. Our single-modality model based on structured inputs predicted low vision prognosis with AUROC of 80% and F1 score of 70%. Our model combining NER text and structured inputs achieved an AUROC of 79% and F1 score of 63%. Explainability studies revealed that important features for predicting low vision prognosis included features clinicians also rely upon, such as best visual acuity and the presence or absence of irreversible ophthalmic findings, as shown in Fig1.

Conclusions : Deep learning models often suffer from a lack of transparency. Our explainability analyses provide insight into the medical context surrounding low vision prognosis and are a vital step towards increasing clinicians’ trust in these models.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

Fig1. Most important model features for predicting low vision prognosis. Global explanations show the top 15 features that contribute most to the prediction for the 3 models. We also show which factors were most important for the prediction of a single patient whose vision did not improve.