Natural-Language Diagnostic Report Generation by Multi-Modal AI for Macular Diseases

Abstract

Purpose : To investigate a multi-modal approach to automatically generate natural-language diagnostic reports for macular diseases, by detecting varied lesions and diseases from color fundus photographs (CFP) and optical coherence tomography (OCT) images and accordingly synthesizing descriptions using natural-language processing (NLP) techniques.

Methods : Three pre-trained deep neural networks were used to detect variable retinal lesions from the CFP / OCT images and predict diagnosis from both modalities independently. A sanity check was performed to ensure the coherence between the detected lesions and the predicted diagnosis. Then a rule-based NLP algorithm descripted the lesions, reported diagnosis and recommended treatment (Fig. 1). A test set of 172 eyes from 127 subjects were successively acquired from July 2020 through September 2020 in our clinic, including 113 normal eyes and 59 abnormal eyes with epiretinal membrane (ERM), dry age-related macular degeneration (AMD), wet AMD and diabetic retinopathy (DR). Each eye consisted of one CFP and 12 radial OCT B-scans. Evaluation of the generated reports was conducted by comparing with the performance of two junior ophthalmologists. A questionnaire was designed and cooperatively judged by two retina specialists to quantitatively grade each report’s readability, correctness of diagnosis and recommendations (Fig. 1). All reports were anonymized to avoid potential bias. Sensitivity and specificity per class was also analyzed.

Results : AI-based NLP reports achieved higher grades in correctness of diagnosis (9.13 vs 9.03 points,) and recommendations (8.55 vs 8.50 points) compared to the junior ophthalmologists, but there was no statistically significance (P=0.43 and 0.85, respectively). For readability, both groups had a satisfactory performance (9.87 vs 9.88 points, P=0.87). Sensitivity of AI-reports and junior ophthalmologists was 0.74 (95% CI, 0.66 - 0.82) and 0.75 (95% CI, 0.67 - 0.83), respectively. Specificity of AI-reports and junior ophthalmologists was 0.94(95% CI, 0.91 - 0.98) and 0.97 (95% CI, 0.95 - 0.99), respectively (Fig. 2).

Conclusions : NLP algorithms-generated diagnostic reports for macular diseases based on multi-modal AI system can achieve similar performance as junior ophthalmologists, suggesting this emerging concept's potential in primary eye service.

This abstract was presented at the 2022 ARVO Annual Meeting, held in Denver, CO, May 1-4, 2022, and virtually.

 

Generation and evaluation for the diagnostic report.

 

Sensitivty and specificity of both groups.