Multimodal Transformer Model to Detect Glaucoma from OCT and Retinal Nerve Fiber Layer (RNFL) Thickness

Abstract

Purpose :
Most prior works using deep learning (DL) models to detect glaucoma were applied to individual imaging modalities, such as fundus photographs, optical coherence tomography (OCT) or visual field (VF). However, a clinician often analyzes multiple sources of data when making a diagnosis. A multimodal model that incorporates multiple types of data may improve glaucoma detection performance. The purpose of this study was to compare a single modality convolutional neural network (CNN), a single modality transformer model, and a multimodal transformer model in detecting glaucoma from raw OCT data and RNFL thickness measurements.

Methods :
A set of 22,464 OCT Spectralis optic nerve head (ONH) circle scans and corresponding RNFL segmentations collected as part of the Diagnostic Innovations in Glaucoma Study (DIGS) and African Descent and Glaucoma Evaluation Study (ADAGES) were analyzed. Data were split on the patient level into training (571 patients, 969 eyes, 20,528 images) and testing (48 patients, 83 eyes, 1936 images) sets. Images were labeled as glaucomatous or healthy based on presence of glaucomatous visual field defects (GVFD). For the single modality models, ResNet50 and Vision Transformer (ViT) architectures were trained solely on OCT RNFL circle scans to detect glaucoma. For the multimodal model, a modified ViT (Figure 1) was trained on unsegmented RNFL circle scans and compared to a vector of 768 thickness measurements derived from the segmented RNFL circle scans to detect glaucoma.

Results :
The ResNet50 trained on RNFL scans had an accuracy of 0.85, area under receiver operating characteristic (AUROC) of 0.91, sensitivity 0.84, and specificity 0.84 (Figure 2). The ViT trained on RNFL scans had an accuracy of 0.85, AUROC of 0.92, sensitivity 0.84, and specificity 0.84. The multimodal ViT trained on RNFL scans and RNFL thickness values had an accuracy of 0.86, AUROC of 0.93, sensitivity 0.84, and specificity 0.84. The transformer based models, ViT and multimodal ViT performed marginally better than the CNN-based model ResNet50. The multimodal ViT performed modestly better than the single modality models.

Conclusions :
A multimodal DL model trained on OCT and RNFL thickness modestly outperforms single modality models trained only on OCT. Multimodal data may provide more accurate deep learning models, and is worth investigating further.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Model architecture.

 

Results on glaucoma detection.