Testing the generalizability of the Hitachi risk score on a different population

Abstract

Purpose : Multiple methods of combining different metrics from Optical Coherence Tomography (OCT) devices have been proposed to increase diagnostic efficacy. The Hitachi paper, published by Fukai et al (2022) describes the comparison of three models developed in Tokyo, Japan. We tested whether the best-performing Hitachi Risk Score model could be applied to a different population.

Methods : The Illinois College of Optometry (ICO) gathered Maestro2 (Topcon Corp., Toyko, Japan) data from patients as part of their routine clinical testing. The data was labeled at the patient level by the clinician using both eyes. Our task was to distinguish the glaucoma patients from the normal and glaucoma suspect patients. This sample of patients was used to evaluate the performance of the Hitachi Risk Score.

The Hitachi models are multivariable logistic regression models based on OCT retinal thickness. The models were developed on a Japanese population (1291 eyes) using Maestro datasets scanned in the vertical direction. Each eye was given a glaucoma risk score, 0 to 100. The higher the risk score, the greater possibility of glaucoma.

The best-performing Hitachi Risk Score model was applied to the ICO 12 by 9 horizontal widefield datasets. Receiver operating characteristic (ROC) curves and area under the curve (AUC) values were generated and compared to individual parameters.

Results : A total of 190 patients were included in the ICO dataset to test the Hitachi Risk Score model. Based on the AUC values, the Hitachi Risk Score model performed higher than the retinal nerve fiber layer (RNFL) and vertical cup to disc ratio (VCDR) with an AUC of 0.88. The RNFL had an AUC of 0.87 and the VCDR had an AUC of 0.76.

Conclusions : When compared to individual parameters, the Hitachi method is generalizable for glaucoma screening. However, the Hitachi method’s AUC decreased from 0.96 on the Hitachi validation set to 0.88 on the ICO dataset and outperformed the RNFL parameter by just 0.01. The current Hitachi Risk Score model was trained on a Japanese population with vertical widefield scans which may explain why the AUC decreased when applied to the ICO datasets. To further analyze the Hitachi method’s performance, individual characteristics of the populations can be compared and more data can be gathered to re-train the algorithm.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.

 

Figure 1 shows the ROC curves of the Hitachi Risk Score model and individual parameters as applied to the ICO dataset.