Investigating the impact of saliency maps on clinician’s confidence in model predictions

Abstract

Purpose : Saliency maps have gained popularity in medical artificial intelligence (AI) studies due to their ability to highlight significant regions for model prediction, allowing greater interpretability of models and raising the prospects of their integration in clinical practice. This study explores whether saliency maps increase clinician confidence in diagnosis via an AI supported workflow.

Methods : 60,133 retinal fundus photography images and labels were gathered from public datasets. These were used to train a classification model on the Google AutoML platform to detect referable and non-referable diabetic retinopathy. Saliency maps were produced using the XRAI algorithm supported by the same platform, providing region-based attribution, suitable for fundus images.
A survey was sent to 6 participants, including 4 ophthalmologists with fellowship-level subspecialty training in retinal disease, and two ophthalmology trainees. They were shown 50 randomly selected images, each repeated 3 times. The image was first presented with the predicted class, secondly by class and Softmax score, thirdly by including a heatmap. After each assistance level, the readers rated their confidence in the prediction via a 5-point Likert scale.

Results : Paired tests showed a significant difference in confidence rating between all assistance types, with saliency maps performing the poorest (3.12±1.28, P<0.001). On a second analysis, we separated predictions into correct and incorrect groups. The unpaired test results show that heatmaps accompanying incorrect labels were associated with a lower confidence than correct ones (incorrect: 2.14±1.15, correct: 3.23±1.25, P<0.001). Similarly, we considered the impact of referable and non-referable cases separately. Results showed no significant difference in confidence from heatmaps between the categories (P=0.097).

Conclusions : Our results show that XRAI heatmaps are associated with lower confidence in model predictions compared to the other two strategies. We emphasize the importance of considering their impact specifically for incorrect predictions and non-referable DR. For the prior, saliency could cause confirmation bias, whereas for the latter, it may mislead if there are no positive findings. Whilst saliency may be of at least some value in model interpretability, for its implementation as diagnostic assistance, higher standards of localisation and meaningful rationale are required.

This is a 2021 ARVO Annual Meeting abstract.