Can a novel comparative AI for diabetic retinopathy based on pairwise comparisons make uncertainty more predictable and explainable?

Abstract

Purpose : Addressing uncertainty and explainability is crucial for ensuring the reliability and trustworthiness of AI systems and for widespread application in clinical practice. We investigated a novel approach, embedding pairwise comparison and ranking of images within an AI environment and applicable to images in diseases with a classification scale.

Methods : We developed and tested an approach, which we term “comparative AI” inspired by the concept of “adaptive comparative judgment. AI models were initially trained using publicly available datasets (Messidor2, APTOS, Kaggle, IDRID) using pairwise comparison to judge which of two images had more severe retinopathy. The system was refined by incorporating Bayesian statistics. The AI was then applied to the DDR dataset by subdividing into validation and testing subsets. DDR images are ranked based on the likelihood of an image being worse/better than specified boundaries. The likelihood provides explainability to the system. We analysed the total uncertainty (both aleatoric and epistemic) by correlating grading errors with ranking and investigated the effect of uncertainty on the AI’s performance.

Results : The test database (1199 images) consisted of 66% no retinopathy, 16% mild, 13% moderate, 3% severe and 3% proliferative. The plot shows the distribution (number and ordinal location) of errors with significant clustering around each of 3 boundaries and normal distributions with truncation. The sensitivity, specificity, positive and negative predictive values were: 0.84,0.89, 64%,96% for mild v moderate; 0.94, 0.89,35%,99.6% for moderate v severe; 0.97, 0.92, 26%,99.9% for severe v proliferative. Other test sets with different prevalence showed similar distributions.

Conclusions : Using comparisons, AI errors are more likely when two images are close together in severity (similar to humans as described by the Weber and Fechner law). Clustering occurs around the boundary, rendering it predictable. We believe the concentration around a classification boundary is an emergent property of comparison-based AI, enabling easy identification and explanations for the AI’s zones of uncertainty. The significance is that such algorithms can be safe (high NPV) and efficient (high PPV) even with modest sensitivity and specificity and highlight cases requiring human judgement.

This abstract was presented at the 2024 ARVO Annual Meeting, held in Seattle, WA, May 5-9, 2024.