Advancing Diabetic Retinopathy Staging with DINOv2: Novel Approaches in Transformer Architecture and Performance Benchmarking

Abstract

Purpose : Our study investigates the staging of diabetic retinopathy (DR) severity utilizing the DINOv2 vision transformer pre-trained on natural images. We propose novel models by fine-tuning DINOv2 for DR classification from color fundus photographs (CFPs) and introduce a novel block expansion strategy (BE) inspired by advancements in the LLM field and benchmark our models against a domain-specific foundation model in ophthalmology (RETFound).

Methods : We employed publicly available datasets with a training, validation, and test split of 70:15:15 across all datasets. For self-supervised (SSL) pre-training on CFPs, images from the AIROGS and Kaggle EyePACS DR challenge were utilized.
We proposed 6 models leveraging the two foundation models. Starting from a CFP foundational model (RETFound) and a model in the natural domain (DINOv2), we fine-tuned them (by training either only the last layer (frozen) or the entire model (unfrozen)) on the given DR datasets. Further, we investigated the impact of fine-tuning DINOv2 on CFP images using the SSL proposed by DINO. In the BE strategy, individual transformer blocks are duplicated and set to 0, with only new blocks being fine-tuned. A description of all models can be found in Table 1.

Results : While training only the last linear layer in DINOv2 (frozen DINOv2, best QKappa of 0.882) resulted in similar performance to training the BB (unfrozen DINOv2, best QKappa of 0.904), for RETFound freezing the BB diminished the performance massively (best QKappa of 0.908 and 0.672 for frozen and unfrozen BB, respectively). Pre-training models on CFPs and freezing the BB during supervised fine-tuning resulted in similar performance to frozen DINOv2 in our study (best QKappa of 0.857 and 0.872 for CFP-DINOv2 and CFP-BE-DINOv2, respectively). For a single dataset our BE strategy massively improved performance. Results across all models and datasets are depicted in Table 2.

Conclusions : ViTs can effectively be adapted for ophthalmic image classification tasks, with our results suggesting that DINOv2 provides general good-performing features for DR grading out-of-the-box, while RETFound requires additional training.

This abstract was presented at the 2024 ARVO Imaging in the Eye Conference, held in Seattle, WA, May 4, 2024.

 

Table 1: Depicts the model names for given architectures, pre-training and training methods and backbone (BB) states.

 

Table 2: Performance of our models (QKappa and rDR (referrable DR) accuracy (across all DR datasets employed.