Abstract
Purpose :
Diagnoses made by Artificial Intelligence (AI) models are taken at face value of the model metrics. There is no existing method to check them except by a specialist. A novel Concept Module was developed to guide the attention of the model using vectorized disease biomarkers given by a specialist. Using this Concept Module, the model could re-confirm or even correct its diagnosis. Evaluation of model performance with and without the Concept Module was done.
Methods :
A Retinal Optical Coherence Tomogram (OCT) image classifier was built on ResNet 18 platform. Concept images were selected by the specialist, and consisted of specific biomarkers of the diseases as seen on OCT. The Concept Module was made by giving these images and random images to a linear classifier to get the coefficients. Once a new image was diagnosed by the model, a concept guided image was created by the Concept Module. The image with concept guided attention was given to the model to predict again.
Results :
Resnet OCT image classifier metrics such as accuracy, precision, recall, F1 score, confusion matrix and AU-ROC curves showed improvement when the model was coupled with the Concept Module. To simulate a real-world scenario, images from a different dataset were also used. The most impressive results were found in diabetic macular edema (DME) images. The model alone predicted 83.13% of these images correctly, but when coupled with the Concept Module, the accuracy increased to 95.33%. A second re-check with the module could improve it further to 98.13%. Model attention shifts when the model corrects its diagnosis (see Image).
Conclusions :
Guiding model predictions using concept guided images from the Concept Module led to improvement in the accuracy of model predictions. Not only does it effectively address the typical decline in model accuracy when deployed in real-world settings, but also demonstrates a substantial improvement as evidenced by the described real-world scenario. The concept module designed by specialists offers additional control over the AI model. By indirectly using a human in the loop, this method ensures AI diagnoses are more accurate and trustworthy. The encouraging results of this novel method can lay the foundation of a credible AI platform for automated diagnosis and help in adoption of AI at a large scale.
This abstract was presented at the 2024 ARVO Imaging in the Eye Conference, held in Seattle, WA, May 4, 2024.