Inter-reader repeatability of structural biomarkers in age-related macular degeneration in the multi-center MACUSTAR study

Abstract

Purpose : Analysis of inter-reader reliability of structural biomarkers for intermediate age-related macular degeneration (iAMD) based on the observational European multi-center study MACUSTAR (ClinicalTrials.gov: NCT03349801).

Methods : For the analysis of the inter-reader repeatability of discrete features (measured by Cohen’s kappa), double gradings of maximum drusen size, presence of pigment epithelium detachment (PED), reticular pseudodrusen (RPD) and vitelliform lesions were performed by two independent readers. The overlap (Dice similarity coefficient [DSC]) of retinal multilayer segmentation was compared between readers as well as readers and a deep-learning (DL) based segmentation model. Mixed-effects models were fitted to data from each retinal layer with the DSC as dependent variable and patient as random intercept term to compare the different pairings (R1-vs.-R2; R1-vs.-DL; R2-vs.-DL), which were included as independent variable.

Results : For the discrete parameters (based on 168 study eyes) almost perfect agreement was obtained for the presence of PED (Cohen’s kappa = 0.87 [0.69–1.00]), while agreement for presence of RPD (0.75 [0.63–0.87]) and vitelliform lesions (0.65 [0.39–0.91]) was substantial and moderate, respectively. For multilayer segmentation, the inter-reader-overlaps (R1-vs.-R2) were comparable between the retinal-nerve fiber layer (RNFL) (DSC estimate [95% CI] of 0.85 [0.80–0.90]), the outer nuclear layer (ONL) (0.82 [0.69–0.94]), the inner photoreceptor segments (IS) (0.78 [0.67–0.90]) and the retinal-pigment-epithelium-drusen complex upper segment (RPEDC-upper) (0.64 [0.52–0.77]) and lower segment (RPEDC-lower) (0.76 [0.66–0.85]). For nearly all retinal layers the reader-DL-segmentation-overlap was overall similar to the inter-reader-overlap (p>0.05). Only the R2-vs.-DL-overlap for the inner plexiform layer (IPL) (+0.03 [0.00–0.06], p=0.04), the RPEDC-upper (+0.07 [0.03–0.11], p=0.002) and the RPEDC-lower segment appear to be minimally worse than the inter-reader-overlap.

Conclusions : Qualitative and quantitative imaging biomarkers exhibited moderate to perfect agreement among readers. Fully-automated DL-based image segmentation provides comparable results in terms of retinal multilayer segmentation. Analysis of inter-reader variability will be important to establish robust structural endpoints for future interventional clinical trials in iAMD eyes.

This is a 2021 ARVO Annual Meeting abstract.