Automated Detection of Malarial Retinopathy-Associated Retinal Hemorrhages

Abstract

Purpose.: To develop an automated method for the detection of retinal hemorrhages on color fundus images to characterize malarial retinopathy, which may help in the assessment of patients with cerebral malaria.

Methods.: A fundus image dataset from 14 patients (200 fundus images, with an average of 14 images per patient) previously diagnosed with malarial retinopathy was examined. We developed a pattern recognition–based algorithm, which extracted features from image watershed regions called splats (tobogganing). A reference standard was obtained by manual segmentation of hemorrhages, which assigned a label to each splat. The splat features with the associated splat label were used to train a linear k-nearest neighbor classifier that learnt the color properties of hemorrhages and identified the splats belonging to hemorrhages in a test dataset. In a crossover design experiment, data from 12 patients were used for training and data from two patients were used for testing, with 14 different permutations; and the derived sensitivity and specificity values were averaged.

Results.: The experiment resulted in hemorrhage detection sensitivities in terms of splats as 80.83%, and in terms of lesions as 84.84%. The splat-based specificity was 96.67%, whereas for the lesion-based analysis, an average of three false positives was obtained per image. The area under the receiver operating characteristic curve was reported as 0.9148 for splat-based, and as 0.9030 for lesion-based analysis.

Conclusions.: The method provides an automated means of detecting retinal hemorrhages associated with malarial retinopathy. The results matched well with the reference standard. With further development, this technique may provide automated assistance for screening and quantification of malarial retinopathy.

Introduction

Malarial retinopathy (MR) is characterized by retinal hemorrhages of varying sizes and shapes, often showing as Roth spots (Fig. 1), retinal whitening, papilledema, and vessel discoloration, ^1–8 and has been shown to be highly sensitive and specific in differentiating cerebral malaria (CM) from other causes of coma in pediatric ⁹ and adult patients. ¹⁰ The hallmark of CM is sequestration of parasites in the vessels of the central nervous system, but postmortem studies have shown that many patients are misdiagnosed. ¹¹ Reasons for this include incidental parasitemia in high-transmission settings and a lack of imaging, laboratory, and electroencephalography facilities in malaria-endemic areas. Ophthalmological expertise that is crucial in diagnosing malarial retinopathy is often also lacking. Malarial retinopathy has great potential, as a surrogate marker for adjunctive therapies, but current methods of quantification and scoring severity of retinopathy are subjective and not evidence-based. ^1,12–14  

The number of retinal hemorrhages in CM correlates with the number of cerebral hemorrhages, so the detection of hemorrhages in MR is a logical first step. ⁶ Therefore, this article primarily focuses on the development of a method for automated detection of MR hemorrhages. 

We have recently developed, studied, and validated retinal image analysis algorithms that are capable of detecting retinal lesions such as hemorrhages, exudates, microaneurysms, drusen, and cotton wool spots, as well as measure retinal arterial and venous parameters in retinal color fundus images, with performance comparable or superior to that of ophthalmologists. ^15–20 We proposed a supervised pixel classification and red lesion detection method based on the analysis of features that include color, shape, and the response of a Gaussian filter bank. ¹⁵ Sinthanayothin et al. ²¹ used a recursive region-growing segmentation method accompanied by a binary thresholding, which detects hemorrhages, microaneurysms, and vasculature from the green channel image and removes vasculature from the final segmentation result. Remaining structures in the image were considered to be hemorrhages and microaneurysms. Gardner et al. presented a supervised algorithm in which a neural network was utilized for the classification of image regions containing hemorrhages and exudates. ²² However, these algorithms are targeted toward the detection of diabetic retinopathy and AMD retinal phenotypes. 

The purpose of the present pilot study is to evaluate an automated method for detecting MR-associated retinal hemorrhages of varying sizes in retinal fundus color images on a dataset of 14 patients (200 fundus images, with an average of 14 images per patient), previously diagnosed with MR. 

Methods

Subjects

Patients were enrolled at Chittagong Medical College Hospital in Chittagong, Bangladesh. Adult patients diagnosed with severe P. falciparum malaria, ¹⁰ based on the presence of asexual stage parasites in microscopy of their peripheral blood, and MR including retinal hemorrhages were included consecutively. All patients or their attending relatives provided written informed consent for participation in a larger study of malarial retinopathy from which the images used in the present study were taken. Retinal images were obtained from (both) eyes of all patients using a handheld retinal camera (Kowa Genesis D; Kowa Optimed Europe Ltd, Sandhurst, Berkshire, UK) through dilated pupils. Image resolution was 1200 × 1600. 

All subjects were treated in accordance with the Declaration of Helsinki. After imaging, images were deidentified and shared with the University of Iowa. The research team at the University of Iowa did not have access to any patient identifiable information. The study was therefore declared exempt by the Institutional Review Board of the University of Iowa. Ethical approval for the larger study on malarial retinopathy was obtained from the Bangladesh Medical Research Council ethics committee and the Oxford Tropical Research Ethics Committee. 

Overview of the Method

The following block diagram (Fig. 2) shows the key steps in the method for detecting MR-associated retinal hemorrhages. 

Figure 1. 

 

Image of a patient with MR with hemorrhages. Notice the Roth spot.

Figure 2. 

 

Block diagram showing the overview of the method.

Reference Standard

A fellowship-trained retinal specialist (MDA), masked to the algorithm results, manually segmented all hemorrhages in all images, using an annotation application (Truthmarker; IDX LLC, Iowa City, IA) that has been previously validated. ²³ Figure 3A shows the retinal fundus image and Figure 3B shows the manual segmentation in action. 

Figure 3. 

 

Reference standard. (A) Fundus image. (B) Expert annotation using Truthmarker. ²³

Hemorrhage Detection Algorithm

Our approach is a splat classification method, which we used previously for retinal image analysis. ^16,24  

Division of a Retinal Image into Splats.

The method performs a watershed segmentation procedure (called “tobogganing”) to generate splats, which initially determines the gradient magnitudes of a grayscale version of a color image, at multiple scales and then utilizes their maximum for segmentation. ^25,26 The maximum of the gradient magnitudes over scales represents a boundary between the contrast structures. The gradient magnitude at a lower scale gives more response at the boundaries of smaller structures, and at a higher scale gives more response at the boundaries of larger structures. Based on the boundaries formed, the image pixels are classified into catchment basin regions by grouping together the pixels with paths of steepest descent terminating at the same local minimum. Thus, the fundus image (Fig. 4A) is divided into a number of regions called “splats” based on region homogeneity. We used a “marker-controlled watershed segmentation method” in which the homogeneous regions in the image are initially marked by foreground markers (single pixel or a group of pixels) using morphological image reconstruction. ²⁷ The regional maxima of the reconstructed image are used to obtain the foreground markers. The background markers are determined from the distance-transform of the binary version of the morphologically reconstructed image. The previously computed gradient magnitude image is modified so that it has regional minima only at the positions of foreground and background markers. Finally, the watershed segmentation of the modified gradient magnitude image is performed. The size of the structuring element selected for the morphological reconstruction controls the size of the marker and hence the splat size. To oversegment, we selected a disk-shaped structuring element with a radius of 1 pixel. The splats generated by the watershed segmentation of the fundus image (Fig. 4A) are shown in terms of splat boundaries overlaid on the image in Figure 4B. The parameter settings result in approximately 4000 splats per image, on average. 

Figure 4. 

 

Retinal image. (A) Fundus image. (B) Image divided into splats.

Splat Feature Extraction and Classification.

For each splat, a set of 43 features was calculated from the pixels within the splat (see Table 1). 

Table 1. 

 

Features Used for MR Hemorrhage Detection

Feature Number	Feature Description
1–6 (6 features)	Mean of pixel values in a splat for RGB and HSV channels.
7–42 (36 features)	Mean of pixel values at the splat boundary for difference of Gaussian image: (Ig−Io); Ig = Gaussian smoothing with σ = 1, 2, 4, 6, 8, 10; Io = original image for each of RGB and HSV channels.
43 (1 feature)	Mean of pixel values in a splat for adaptive histogram equalization image.

We chose features that represent the color as well as its variation across the fundus image. Some of the features in this set were used previously. ¹⁶ A subset (features 1–6) of the feature set accounts for the mean color within a splat in red, green, blue (RGB) and hue, saturation, value (HSV) channels. The other subset (features 7–42) characterizes the intensity variations across the fundus image in terms of difference of Gaussians at various scales (σ). We introduced a feature (43) of the mean intensity within a splat in an adaptive histogram equalization image, which brings out the structural details due to the local contrast, at various scales. The automated feature selection (sequential forward feature selection) decreased the classification performance based on the area under the receiver operating characteristic (ROC) curve, and hence the complete feature set of 43 features was used. 

A classifier was then trained on the feature set of each splat from a training set of images. We used a linear k-nearest neighbor (kNN) classifier ^28,29 based on the Euclidean distance measurement for the supervised splat classification. The feature set of 43 features extracted from each of the splats in a training image, as above, represents the position of that splat in a 43-dimensional feature space (training data point). The classifier is trained by associating the feature set with the label associated with the splat (hemorrhage or nonhemorrhage), derived from the reference standard created by the expert, as above. In the reference standard, a splat was labeled as a hemorrhage splat if more than 50% of the splat area was segmented manually. 

In the testing phase, a feature set is extracted as above from each splat in the test image (Fig. 5A). The previously trained kNN classifier is then queried with the feature set, and provides the labels of kNNs in the 43-dimensional feature space in return. The fraction of the kNNs labeled as hemorrhage splat provides the likelihood, which is assigned to the test splat. This likelihood, a number between 0–1, is then assigned to all pixels belonging to the test splat; and a hemorrhagic map can thus be created, indicating how likely each splat is to be a part of the hemorrhage (Fig. 5B). Therefore, the more similar the features of a test splat to a hemorrhage, the higher the likelihood of the splat being inside the hemorrhage assigned to it (a number closer to 1). Based on the higher likelihood value, the splat may be differentiated with respect to its background. The likelihood map can be thresholded at various values between 0 and 1, producing different system sensitivities and specificities, and thus producing a receiver operating characteristic (ROC) curve. The optimal value of k for kNN classifier was determined based on area under the ROC curve. For this experiment, the maximum area under the curve (AUC) was found corresponding at k = 181. 

Figure 5. 

 

Test image. (A) Fundus image with the overlaid splat boundaries. (B) Hemorrhage likelihood map: bright pixels indicate higher likelihood of the splat being a hemorrhage.

Analysis

Performance of our algorithm was measured in a crossover design experiment. The dataset of all patients was partitioned into two complementary subsets, two patients for testing and the remainder for training the classifier; and this process was then repeated 13 times for different permutations of the training set. 

The performance was determined for splat-based as well as hemorrhage lesion-based analysis. The splat-based analysis measures how well the system can classify a splat as part of a hemorrhagic or nonhemorrhagic region, whereas the lesion-based analysis measures system performance in terms of classifying a retinal lesion as a hemorrhage or nonhemorrhage. 

The following metrics were used: 

1.  
   True positive (TP) splats (or lesions): The number of splats (or lesions) marked by both our algorithm and the expert as hemorrhage.
2.  
   False negative (FN) splats (or lesions): The number of splats (or lesions) marked as hemorrhagic by the expert but not by our algorithm.
3.  
   True negative (TN) splats (or lesions): The number of splats (or lesions) marked as nonhemorrhagic by both our algorithm and the expert.
4.  
   False positive (FP) splats (or lesions): The number of splats (or lesions) marked by our algorithm as hemorrhagic, but not by the expert.

Sensitivity was defined as the fraction of hemorrhage splats (or lesions) marked by the expert (TP + FN) that were detected as hemorrhage splats (or lesions) by the algorithm (TP; Eq. 1), while specificity was defined as the fraction of nonhemorrhage splats (or lesions) marked by the expert (TN + FP), that were detected as non-hemorrhage splats (or lesions) by the algorithm (TN; Eq. 2). The hemorrhage lesion was considered “detected” if at least one of the hemorrhage splats marked by the expert was detected by the algorithm. 

 

 

 

 

 

 

The total number of TP, FN, TN, and FP values for the 14 test sets, and the respective sensitivity and specificity measures, were evaluated by varying the threshold on the hemorrhage likelihood map. The sensitivity and specificity values obtained at the likelihood threshold of 0.12 are reported in Table 2 for splat-based analysis, whereas the sensitivity and false positive values obtained at the likelihood threshold of 0.38 are reported in Table 3 for lesion-based analysis. For the splat-based analysis, an ROC curve was determined (Fig. 6A), whereas for the lesion-based analysis, a free-response operating characteristic (FROC) curve was obtained (Fig. 6B), using an online ROC analysis tool. ³⁰ The FROC curve describes the actual performance of the hemorrhage lesion detection and plots the sensitivity of the proposed method with respect to all hemorrhage lesions in the test set against the average number of false positives detected per image. 

Figure 6. 

 

Hemorrhage detection performance. (A) ROC curve: splat-based analysis. (B) FROC curve: lesion-based analysis.

Table 2. 

 

Performance of the Method for Hemorrhage Detection in Terms of Splats, at Likelihood Threshold of 0.12

Total hemorrhage splats	454,581	Sensitivity (%)	80.83
False negative splats	87,143
Total nonhemorrhage splats	1,862,830	Specificity (%)	96.67
False positive splats	62,032
AUC	0.9148	95% CI for AUC	0.8254–0.9842

Table 3. 

 

Performance of the Method for Hemorrhage Detection in Terms of Lesions, at Likelihood Threshold of 0.38

Total hemorrhage lesions	917	Sensitivity (%)	84.84
False negative lesions	139
False positive lesions	1079	False positive lesions/image	3
AUC	0.9030	95% CI for AUC	0.8724–0.9325

The ROC curve and AUC were reported for the sample of 14 patients included in the study. The AUC value is not a true value but the sample value with statistical error and may vary for different samples. ³¹ Therefore, we reported a range of AUC values (confidence interval: CI) within which a true value lies with a certain degree of confidence. We included a 95% confidence interval with the ROC curve representing the range of values in which the true value lies. 

Results

For the splat-based analysis, the sensitivity and specificity values obtained from the ROC curve at the likelihood threshold of 0.12 are 80.83% and 96.67%, respectively (see Table 2). For the lesion-based analysis, the FROC curve shows the sensitivity of 84.84% against the average of three false positives detected per image, at the likelihood threshold of 0.38 (see Table 3). The respective ROC and FROC curves are shown in Figure 6. The AUC and the associated 95% CI are on a splat basis, 0.9148 and 0.8254–0.9842, respectively, and on a lesion basis, 0.9030 and 0.8724–0.9325, respectively. The average number of hemorrhages per patient detected by the system was estimated to be four or more out of every five hemorrhages annotated in the reference standard. The running time required per image was 10 minutes when processed on a standard personal computer with dual-core processor (Intel Core 2 Duo; Intel, Santa Clara, CA) running at 3 GHz. 

Discussion

Our results demonstrate that the automated method is capable of detecting retinal hemorrhages on retinal fundus color images, and match well with the reference standard as shown by the area under the FROC curve (AUC = 0.9030). The FROC curve and the corresponding AUC indicate the ability of the system to distinguish between hemorrhagic and nonhemorrhagic regions. This means that, given a randomly selected retinal region in the image, the automated system will correctly detect in 90.3% of cases the presence or absence of a hemorrhage in that region. Note that the reported lesion detection sensitivity of 84.84% is obtained at the cost of an average of three FPs per image, which may be a high FP rate for the application where the automated system is used to assist image analysis by a human. 

The hemorrhages may possess color features similar to the surrounding blood vessels, as well as the blood-perfused retina. Thus, the retinal regions with similar features to the hemorrhages may be detected as false positives. This may be evident from the literature, ¹⁵ where per-lesion sensitivity and the respective average number of FPs per image were compared for number of lesion detection methods, along with a human expert. Most of these methods resulted into an average of three or more FPs per image at the average per-lesion sensitivity of 47%, including 2.6 FPs per image at the per-lesion sensitivity of 69% by a human expert. Relative to these reports, we presented the per-lesion sensitivity of 84.84% at the average of three FPs per image. To further improve the hemorrhage detection sensitivity and reduce the number of FPs, the method may require an increased number of relevant features, as well as the increased length of training and test datasets with a wider variety of hemorrhages observed in MR cases. 

Potentially, the use of automated techniques such as this, providing quantitative estimates of the signs of MR with good reproducibility may improve the objectivity of scoring and classification of malarial retinopathy. This technique has the potential for assisting with the assessment of malarial retinopathy in both research and clinical settings. ³²  

The varying shapes and sizes of hemorrhages were modeled in terms of splats, which enabled the splat-specific feature extraction and classification. This may decrease the computation time required by pixel-based classification and may prevent the inclusion of pixel level noise into the feature set as compared with the pixel processing based methods described in the literature. ^15,21  

The methods by Sinthanayothin et al. ²¹ and Gardner et al. ²² reported to achieve the sensitivity of hemorrhage detection as 77.5% and 73.8%, respectively, as compared with the sensitivity of 84.84% obtained by the proposed method. The specificity of 88.7% reported in Sinthanayothin et al. ²¹ was obtained by the validation based on a 10 × 10 pixel grid overlaid on the image. An ophthalmologist manually filled the grid area containing hemorrhages and microaneurysms, which was then compared with automatically identified grid area for the location of lesions. When analyzing the performance of a hemorrhage detection method on a set of images, the number of TN regions may always be high compared with the TP regions. Thus, the specificity may always remain high, which does not represent the real detection performance of the system. Therefore, it may be advisable to present the average number of false positive detections per image at the respective sensitivity, as reported for the proposed method. 

This study has a few limitations worth mentioning. The technique is currently limited only to retinal hemorrhages; and when used alone, these have poor specificity for severe malaria. ¹⁰ This is likely to be because they are not a result of the core pathological process of microvascular obstruction in severe malaria, ³² and not a feature unique to severe malaria. The current version of the software attempts to demonstrate the initial step toward the automated detection of MR-associated retinal pathologies such as hemorrhages, against the current methods of subjective clinical analysis. Future expansion of automated image assessment to other unique and specific features of MR such as retinal whitening, vessel discoloration, ^1,4,5,13 and papilledema is ongoing. 

The running time of the system may be considered high due to the small splat size. We chose the splat size depending upon the smallest hemorrhage size in the dataset. The smaller the size of a splat compared with that of the smallest hemorrhage, the more accurate the detection performance of the system may be. This is due to the ability of smaller splats to demark the hemorrhage boundaries more accurately compared with the larger ones. If the average splat size is greater than the hemorrhage size, it may include the surrounding background features into a feature space and may be classified erroneously. As the splat size decreases, the average running time increases due to an increase in the number of splat feature extractions and classifications. Most of the running time was spent in the splat classification process. Future refinements to reduce this running time are needed so it can deal with a large numbers of images rapidly. 

Finally, training and test datasets consisted of images exclusively containing retinal hemorrhages. Though the system specificity for nonhemorrhagic regions is high, it may be advisable to include more test images with no hemorrhages, in order to avoid bias in the dataset. Our future efforts will focus on analyzing larger datasets to provide a more robust assessment. 

In summary, we developed an automated MR hemorrhage detection method based on splat classification and validated it on a dataset of patients previously diagnosed with MR. The results were validated with the reference standard showing the potential of the method in providing diagnostic assistance in detection of MR. 

Footnotes

 Supported by the Veterans Administration Center of Excellence for Prevention and Treatment of Visual Loss; NIH grants EY017066 and EB004640; and the University of Iowa and Research to Prevent Blindness. The data collection was supported by a Research Fellowship from the British Infection Society (RJM) and the Wellcome Trust of Great Britain (grant number 077166/Z/05/Z).

Footnotes

 Disclosure: V.S. Joshi, None; R.J. Maude, None; J.M. Reinhardt, None; L. Tang, None; M.K. Garvin, None; A.A. Sayeed, None; A. Ghose, None; M.U. Hassan, None; M.D. Abràmoff, None