Automatic Identification of Reticular Pseudodrusen Using Multimodal Retinal Image Analysis

Abstract

Purpose.: To examine human performance and agreement on reticular pseudodrusen (RPD) detection and quantification by using single- and multimodality grading protocols and to describe and evaluate a machine learning system for the automatic detection and quantification of reticular pseudodrusen by using single- and multimodality information.

Methods.: Color fundus, fundus autofluoresence, and near-infrared images of 278 eyes from 230 patients with or without presence of RPD were used in this study. All eyes were scored for presence of RPD during single- and multimodality setups by two experienced observers and a developed machine learning system. Furthermore, automatic quantification of RPD area was performed by the proposed system and compared with human delineations.

Results.: Observers obtained a higher performance and better interobserver agreement for RPD detection with multimodality grading, achieving areas under the receiver operating characteristic (ROC) curve of 0.940 and 0.958, and a κ agreement of 0.911. The proposed automatic system achieved an area under the ROC of 0.941 with a multimodality setup. Automatic RPD quantification resulted in an intraclass correlation (ICC) value of 0.704, which was comparable with ICC values obtained between single-modality manual delineations.

Conclusions.: Observer performance and agreement for RPD identification improved significantly by using a multimodality grading approach. The developed automatic system showed similar performance as observers, and automatic RPD area quantification was in concordance with manual delineations. The proposed automatic system allows for a fast and accurate identification and quantification of RPD, opening the way for efficient quantitative imaging biomarkers in large data set analysis.

Introduction

Age-related macular degeneration (AMD) is a progressive eye disease affecting mainly the elderly and causing vision loss at advanced stages.¹ The early stages of AMD are characterized by the presence of pigmentary changes and drusen, which are deposits accumulating between the retinal pigment epithelium (RPE) and the Bruch's membrane. A newly appreciated extracellular lesion in AMD, commonly termed “subretinal drusenoid deposits” or “reticular pseudodrusen” (RPD), presents different characteristics and distribution than normal drusen and is a strong risk factor for progression to advanced AMD.^2–7 Therefore, its identification and quantification is of paramount importance for a better understanding of disease progression. 

Reticular pseudodrusen are visible on color fundus (CF) photography, fundus autofluorescence (FAF) imaging, and near-infrared (NIR) imaging among other retinal imaging modalities such as confocal blue reflectance, indocyanine green angiography, spectral-domain optical coherence tomography (SD-OCT), and fluorescein angiography.^8–13 On CF images, RPD are described as indistinct, yellowish interlacing networks with a width of 125 to 250 μm.¹⁴ On FAF images, RPD are characterized as hypofluorescent lesions, while on NIR images, RPD are characterized as groups of hyporeflectant lesions against a mild hyperreflectant background.^15–17 Previous studies^12,17 have reported a difference in sensitivities for RPD detection among image techniques. However, RPD identification using a single-image modality is challenging, as the characteristic changes associated with RPD are often subtle and may not always be detected when using only one imaging technique. Therefore, for an accurate diagnosis, RPD detection should be performed with two or more image modalities.¹¹ Although other studies have investigated and compared the performance of individual image techniques for RPD detection,^12,17 a study of the performance obtained by using multiple image modalities simultaneously has not been performed yet, to the best of our knowledge. 

Despite its expected higher accuracy, grading of multimodality images represents a considerable workload for a human grader. Machine learning algorithms have huge potential for dealing with complex information extracted from different image modalities. Furthermore, automatic systems are not influenced by fatigue and mindset and, therefore, are less prone to variability than humans. Previously developed systems for the automatic detection of drusen showed good performance on CF images.^18–21 Whether they also perform well fusing information from different image modalities is currently unknown. To the best of our knowledge, there is no method for the automatic identification of RPD fusing information from different image modalities. 

The aim of the present study was 2-fold. Firstly, we evaluated the performance and the agreement between human observers by using single- as well as multimodality grading approaches for RPD detection. In the single-modality approach, RPD detection was performed by using only one image technique (namely, CF, FAF, or NIR). In contrast, during the multimodality grading session, the observers evaluated the three available image modalities simultaneously. Secondly, we aimed to investigate the effectiveness of a novel machine learning algorithm for the automatic identification and quantification of RPD by using combined information from different image modalities and comparing its performance to that of human observers. 

Methods

Study Data Set

A set of subjects with and without RPD was selected from the Rotterdam Study, a prospective cohort study investigating risk factors for chronic diseases in the elderly.²² The study adhered to the tenets set forth in the Declaration of Helsinki, and investigational review board approval was obtained. Only patients with CF, FAF, and NIR images available were included in this study. Color fundus images were taken by using a 35° field-of-view Topcon TRC 50EX fundus camera (Topcon Optical Company, Tokyo, Japan) with a Sony DXC-950P digital camera with a resolution of 768 × 576 pixels (Sony Electronics, Inc., New York, NY, USA). Fundus autofluorescence and NIR images were taken with a Heidelberg Retina Angiograph 2 (Heidelberg Engineering, Heidelberg, Germany) with a field of view of 30° and a resolution of 768 × 768 pixels. In total 278 eyes of 230 patients aged 65 years and older were selected from the last examination round of the Rotterdam Study. All CF images were graded according to the Wisconsin Age-Related Maculopathy Grading²³ and the International Classification and Grading System for Age-Related Maculopathy and Age-Related Macular Degeneration²⁴ by local graders of the Rotterdam study, using visual assessment. These annotations constituted the reference standard for our study. We selected all the eyes for which RPD were identified in this round (N = 72) from CF images. Status of RPD was also confirmed on FAF and NIR imaging. For positive and negative controls, we selected eyes that were graded by the local Rotterdam study graders as having soft distinct or soft indistinct drusen but without RPD (N = 108) and eyes that did not contain any type of drusen (N = 98), respectively. The positive and negative controls did not have any signs of RPD in the other modalities (FAF and NIR). As the database did not contain any information about the extent of RPD area, two human observers (G.H.S.B., C.B.) made RPD area delineations in consensus by using the three modalities simultaneously for the eyes containing RPD. These delineations were used as reference standard for the quantification of RPD area. 

Observer Study: Single- Versus Multimodality Grading

All images were evaluated independently by two human observers (G.H.S.B., C.B.) for evidence of RPD. Reticular pseudodrusen were defined as indistinct, yellowish interlacing networks with a width of 125 to 250 μm on CF images¹⁴; groups of hyporeflectant lesions in regular patterns on FAF images^15–17; and groups of hyporeflectant lesions against a mildly hyperreflectant background in regular patterns on NIR images.¹⁷ Observer 1 had 4 years of reading experience for all three imaging modalities, whereas observer 2 had 19 years of reading experience on CF imaging and 5 years on FAF and NIR imaging. The observers were asked to use a scoring system ranging from 0 to 1, indicating the likelihood of presence of RPD. Two different grading approaches were used: single- and multimodality grading. During single-modality grading, the observers graded each image modality separately in a randomized order. Color fundus, FAF, and NIR images were pooled and shown randomly to the observers. Observers were also asked to indicate whether the image was of sufficient quality for grading. The quality of an image is deemed insufficient when it becomes difficult or impossible to make a confident assessment regarding the presence of RPD. During multimodality grading, observers were asked to diagnose RPD after observing CF, FAF, and NIR images from the same eye simultaneously. The eyes were shown in randomized order in this grading session as well. 

In a separate grading session, the observers manually delineated in consensus the area covered by RPD, based on one single modality, that is, single-modality RPD delineation on CF, FAF, or NIR images. Only the 72 eyes containing RPD as indicated by the reference were taken into account for the quantification of RPD area. 

Automatic Reticular Pseudodrusen Identification

The proposed machine learning algorithm simultaneously analyzed the available modalities from an eye examination to automatically identify reticular pseudodrusen areas. The algorithm assigned the complete eye examination a probability between 0 and 1, indicating the probability of presence of RPD and providing a quantification of the area covered by the lesions. To accomplish this, the algorithm performed three steps: preprocessing, feature extraction, and classification and quantification. 

Preprocessing.

In the preprocessing step, two different methods were applied to the images: image registration and vessel removal. 

1.  
   Registration will provide a geometric alignment across modalities to identify corresponding pixels that represent the same scene. This multimodal image registration was performed by using a semiautomatic affine method, where the images are deformed to accurately match user-specified points or landmarks.²⁵ In this study, three corresponding landmarks on prominent image locations, such as vessel bifurcations, were manually selected on each modality and used to perform the registration.
2.  
   To reduce intensity variations due to presence of vessels, the retinal vasculature was removed from the images. The vasculature was automatically extracted by using a previously developed algorithm²⁶ and used as input in an inpainting algorithm,²⁷ which removes the vessels by interpolating intensities at the supplied image locations. Figure 1 shows an example CF image (Fig. 1A), FAF image (Fig. 1B), and NIR image (Fig. 1C) of an eye and Figures 1D through 1F shows their corresponding images after vessel removal, respectively.

Figure 1

 

Coregistered (A) CF, (B) FAF, and (C) NIR images and their corresponding results after vessel removal (D–F).

Feature Extraction.

To perform an automatic analysis of the images, the machine learning algorithm uses information that is extracted from the images and encoded in numerical values or so-called features. To do so, each color channel of the CF image, as well as the FAF and NIR image, was separately convolved with a set of Gaussian filters. These filters are based on Gaussian derivatives up to second order at different scales and are invariant to rotation and translation.²⁸ For each resulting filtered image, the mean, standard deviation, skewness, and kurtosis values in a circular neighborhood around each pixel were calculated. The corresponding features for each pixel were then obtained by concatenating these extracted values in a single feature vector. 

Classification and Quantification.

To determine whether a pixel is part of an RPD area, a random forest classifier was used to obtain an automatic classification based on the calculated features. This classifier operates by constructing a multitude of decision boundaries (trees) to make a separation between multiple classes.²⁹ After training, the random forest classifier provided a probability between 0 and 1 indicating the probability that the pixel belongs to an RPD area, based on labeled training examples and the input pixel feature vector. Figure 2 shows the images of the modalities of an example eye (Figs. 2A–C) and the output of the classifier (Fig. 2D). Finally, an image score indicating the likelihood of the eye examination to contain RPD was assigned by taking the 99th percentile of the obtained probability map. 

Figure 2

 

Example of the classification result obtained by the proposed machine learning algorithm. Given that an eye examination consisted of (A) a CF image, (B) an FAF image, and (C) an NIR image, the algorithm outputs (D) a probability map indicating the likelihood for each pixel to be part of an RPD area. Red values indicate higher probability to be RPD.

To quantify the area covered by RPD, a threshold was set on the probability map. This threshold was image based and experimentally determined as the 55th percentage of the maximum value of the probability map. Only the area inside the Early Treatment Diabetic Retinopathy Study (ETDRS) grading grid was taken into account for the quantification. 

Statistical Analysis

The performance of the observers and the proposed machine learning algorithm for the single- and multimodality approaches was evaluated by measuring the area (Az) under the receiver operating characteristic (ROC) curve.³⁰ Statistical comparisons were made by using bootstrap analysis with 5000 bootstraps.³¹ Bootstrap analysis is an nonparametric test that is commonly used to estimate the variance of ROC analysis. Results with a P value lower than 0.05 were seen as statistically significant. Bonferroni correction was applied to counteract the problem of multiple comparisons.³² For observers, κ statistics were also reported to assess interobserver variability.³³ As the proposed machine learning algorithm requires labeled example data for training, the evaluation was performed by using a patient-based leave-one-out strategy.³⁴ 

Automatic quantification of RPD area was evaluated by calculating the percentage of detected RPD area inside the ETDRS grading grid and was compared with the observer delineations. The RPD area agreement with observers was measured by using intraclass correlation (ICC) statistics. 

Results

Image Quality Assessment

Table 1 shows the image quality analysis of the observers for the different image modalities. Of the 278 eyes, only 172 (61.9%) were graded by both observers as having all image modalities with good quality and were established as the “good quality” set for the subsequent data analysis. Bad quality of the FAF image was the main reason for a bad-quality indication for the multimodal examination (CF+FAF+NIR). 

Table 1

 

Number and Percentage of Good-Quality Images as Indicated by Observers for the Different Image Modalities Independently

	Observer 1	Observer 2	Consensus
CF	272 (97.8%)	268 (96.4%)	264 (95.0%)
FAF	211 (75.9%)	195 (70.1%)	185 (66.5%)
NIR	269 (96.8%)	265 (95.3%)	264 (95.0%)

 Last column shows the number of images for which both observers agree that the image is of good quality.

Comparison of Single- and Multimodality Grading

Figure 3 shows the ROC curves for the single modality approaches using CF (Fig. 3A), FAF (Fig. 3B), or NIR (Fig. 3C) images and the multi modality grading approach (Fig. 3D). The point on the curve closest to the upper left corner in the ROC curve is used to compute sensitivity/specificity pairs. 

Figure 3

 

Receiver operating characteristic curves for the identification of eyes with RPD using (A) CF images, (B) FAF images, (C) NIR images, and (D) a multimodality setup.

Table 2 shows the Az values and sensitivity/specificity pairs for the single- and multimodality grading of observer 1 and observer 2, respectively, calculated on the full data set and on the subset of good-quality images as indicated by both observers. The performance of both observers for RPD detection considerably increased when performing a multimodality grading. 

Table 2

 

Performance of Observer 1 and Observer 2 for RPD Detection Using Single- and Multimodality Grading

	All			Good Quality
	Az	Se	Sp	Az	Se	Sp
Observer 1
Single modality
CF	0.879*	0.778	0.951	0.888*	0.789	0.974
FAF	0.881*	0.889	0.806	0.959	0.946	0.966
NIR	0.936	0.903	0.956	0.936	0.929	0.918
Multimodality	0.940	0.917	0.961	0.956	0.944	0.963
Observer 2
Single modality
CF	0.944	0.944	0.989	0.944	0.930	0.917
FAF	0.793*	0.653	0.951	0.961	0.973	0.946
NIR	0.932	0.903	0.922	0.929*	0.900	0.918
Multimodality	0.958	0.972	0.942	0.974	1.000	0.949

 Area under the ROC values and optimal sensitivity (Se) and specificity (Sp) values are reported.

*  Indicates a statistical significant difference of the Az value with respect to the multimodality approach.

κ statistics were calculated to measure interobserver variability during single- and multimodality grading sessions. Table 3 shows the κ values between the observers for the different grading sessions. Observers achieved a higher agreement with multimodality grading. When considering only good-quality images, observers also achieved high agreement when using FAF images. 

Table 3

 

κ Agreement and 95% Confidence Intervals Between Observers for Single- and Multimodality Reading Sessions

	All		Good Quality
	κ	95% CI	κ	95% CI
Single modality
CF	0.654	(0.556–0.752)	0.724	(0.632–0.817)
FAF	0.468	(0.363–0.572)	0.938	(0.879–0.998)
NIR	0.884	(0.822–0.945)	0.839	(0.767–0.910)
Multimodality	0.911	(0.857–0.965)	0.936	(0.874–0.998)

 CI, confidence interval.

Performance of the Automatic Method

The ROC curves for the proposed machine learning algorithm are shown in Figure 3. The corresponding Az values and the sensitivity/specificity pairs for the single- and multimodality approaches are summarized in Table 4. 

Table 4

 

Performance of the Automatic System for RPD Detection Using Single- and Multimodality Grading

	All			Good Quality
	Az	Se	Sp	Az	Se	Sp
Single modality
CF	0.942	0.833	0.922	0.939	0.887	0.860
FAF	0.844*	0.806	0.747	0.935	0.919	0.891
NIR	0.927	0.847	0.893	0.919	0.814	0.902
Multimodality	0.941	0.875	0.873	0.949	0.861	0.882

 Area under the ROC values and Se and Sp values are reported.

*  Indicates a statistical significant difference of the Az value with respect to the multimodality approach.

Quantification of the Area Covered by RPD

The box plots in Figure 4 show the RPD area percentage inside the ETDRS grading grid as delineated by the observers and as identified by the automatic system. Only eyes that were of good quality as indicated by both observers were taken into account. The multimodality area delineations made during consensus grading of the two observers was used as the reference for the RPD area quantification. 

Figure 4

 

Box plots showing the percentage of RPD area inside the ETDRS grading grid. Manual multimodality was seen as the reference and is shown in gray. Single-modality manual area percentages are shown in blue. The area percentages obtained by the automatic method for single- and multimodality analysis are shown in red.

The agreement between single-modality RPD area delineations made by the observers and the reference delineations set by using multimodal information reached ICC values of 0.580 (−0.034; 0.830), 0.790 (0.409; 0.920), and 0.930 (0.763; 0.976) for the CF, FAF, and NIR delineations, respectively. For the automatic quantification of the RPD area, ICC values of 0.637 (0.395; 0.796), 0.389 (0.082; 0.631), and 0.557 (0.280; 0.747) were obtained for the single-modality analysis of CF, FAF, and NIR images with respect to the reference delineations. Comparing the automatic multimodality approach with the reference standard, an ICC value of 0.704 (0.495; 0.837) was obtained. 

Discussion

In this study, we assessed the performance achieved for RPD detection by using multimodal information and compared it to the one obtained by using several single-image techniques. In our larger data set,^11–13,17 we have demonstrated that a significantly higher performance, as well as a better interrater agreement, is achieved when the reticular pattern is assessed in a multimodality grading approach. Moreover, our automatic machine learning algorithm for RPD detection and quantification using multimodal information performed within the same range as the human graders. 

Two independent human observers identified RPD areas by using two different grading protocols. During the single-modality grading session, only information from a single-image technique was available, whereas during the multimodality approach, the observers evaluated evidence of RPD by using all the modalities simultaneously. Both observers achieved higher performance with the multimodality approach, reaching Az values of 0.940 and 0.958 (Fig. 3; Table 2). Although previous studies^8,11,12 have evaluated the accuracy for detecting RPD of single-image modalities only, our results confirm their conclusions that a more accurate diagnosis of RPD is obtained by using multiple image modalities. 

In contrast to observer 1, observer 2 achieved high performance on RPD assessment with CF images. Possible reasons for this observation include the vaster experience of this observer on this modality and the lower sensitivity of this image technique.¹² The disparity between the observers' performance was substantially reduced when the assessment was performed by using multiple image modalities (Table 2). When the observers scored FAF images, the performance was significantly lower than when they used multimodality reading. This may be due to the poor quality level of the FAF images. Only 66.5% of the images were considered of good quality by both observers, as shown in Table 1. During FAF acquisition, a mean intensity image is constructed to reduce noise in the image. However, eye movements may cause displacement errors, resulting in a lower contrast and thus lower quality of the FAF image. Another reason is the presence of cataracts in the study population. The wavelength used for FAF imaging is affected more by cataracts than the one used in NIR imaging, resulting in lower image quality. As shown in Table 2, the adoption of a multimodality grading approach can overcome image quality issues, maintaining a high detection performance independently of the quality level of a particular image technique. When considering only the subset of good-quality images, the performance of both observers increased for both single- and multimodality gradings. 

Interobserver agreement was also investigated by using the two grading protocols. Table 3 shows that the agreement between observers substantially increased when multiple imaging techniques were used to evaluate the evidence of RPD. When taking only the subset of good-quality images into account, the agreement between observers improved when using CF, FAF, and the multimodal approach. However, agreement when using CF images is still substantially lower than when using the other modalities. Other studies¹² have included multiple graders but no information about interobserver agreement has been reported. 

In this study, we also developed and evaluated a machine learning algorithm for the automatic identification and quantification of RPD using multimodal information. The results showed that the proposed system achieved similar performance as the observers (Fig. 3; Table 4). Similar to the observers' gradings, the incorporation of multimodal information improved the performance of the algorithm. Using multimodal information, the proposed algorithm achieved an Az value of 0.941 and a sensitivity/specificity pair of 0.875/0.873. Compared to the observers, who reached a κ agreement of 0.87 with the reference, the automatic system had a κ agreement of 0.70. However, 20% of the misclassified cases correspond to cases where there was disagreement between the observers. Of the false-positive cases, 9 cases contained low-quality images, 3 cases presented geographic atrophy, 1 case showed a neovascular macular detachment, and 12 cases contained soft indistinct drusen. As described in other publications,^11,35,36 RPD and drusen have very similar characteristics and they might therefore more easily be misinterpreted by the automatic system. Better discriminant features, such as image context information or local intensity changes, might improve the performance of the automatic system, but this has to be further investigated. 

Quantification of RPD area is a more difficult task owing to the undefined boundaries of RPD.^15–17 When comparing the manual delineations performed on CF images with the reference delineations based on multimodal information, an ICC value of 0.580 was achieved. When comparing the FAF or NIR delineations with the reference delineations, the agreement was better, reaching ICC values of 0.790 and 0.930, respectively. As presented in Figure 4, the RPD area was underestimated when using CF images as compared with the other image techniques. As reported in previous publications,^12,17 the visibility of RPD differs over imaging modalities, causing these differences. As RPD are more pronounced on FAF and NIR, the delineations on these modalities were more similar to reference delineations. The quantified RPD area, which was automatically obtained by the proposed algorithm, was in agreement with the area delineated by the observers, reaching an ICC value of 0.704. Of note, only images of good quality were used for RPD area quantification because images with insufficient quality were not suitable, as it was nearly impossible for observers to delineate RPD area on these images. Another limitation of this study was that the multimodal approach included only fundus images, excluding information obtained with SD-OCT. Including this modality in the multimodal protocol might result in better understanding of the reticular pattern, thus increasing accuracy in their identification.^9,10,35 Spectral-domain OCT can provide 3-D information about RPD formation and is essential for RPD volume measurements. This enhancement will be of great importance for clinical trials studying the development and progression of RPD. We will investigate this improvement in further studies. 

In conclusion, we were able to show that a multimodal approach significantly increased observer performance and interobserver agreement for detection of RPD in fundus images when the information of different imaging modalities was evaluated simultaneously. Furthermore, an automatic machine learning algorithm for detection and quantification of RPD using multimodal information was developed and evaluated, showing comparable results with those obtained by observers. The area covered by RPD was also automatically quantified by the algorithm, tallying the values manually provided by the observers. The absence of SD-OCT is regarded as a limitation of this study and will be investigated in future work. This automatic algorithm yields a quick and reliable diagnosis and quantification of RPD, for large data set analysis within population studies and for gaining insights into risk factors involved in AMD and disease progression. 

Acknowledgments

Supported by a ZonMw grant: “A cost-effective solution for the prevention of blindness using computer-aided diagnosis and fundus photography,” with project number 11.631.0003. 

Disclosure: M.J.J.P. van Grinsven, None; G.H.S. Buitendijk, None; C. Brussee, None; B. van Ginneken, None; C.B. Hoyng, None; T. Theelen, None; C.C.W. Klaver, None; C.I. Sánchez, None.