Retinal Hemorrhage Labeling: Implications for Machine Learning Study Design

Abstract

Purpose : To assess utility and limitations of ophthalmologist labeling of fundus photos of retinal hemorrhage (RH) as a preliminary step in development of a machine learning algorithm to interpret RH findings in children.

Methods : To initiate training of a machine learning algorithm for interpretation of RH findings in young children, 4 pediatric ophthalmologists labeled 30 fundus photos from adults and children with RH due to medical conditions (e.g., leukemia, increased intracranial pressure, retinal arterial macroaneurysm, retinal vein occlusion), accidental or abusive head trauma, or vaginal birth. In an effort to maximize clinical relevance, ophthalmologists made the labeling independently without preparatory discussion or explicit consensus of label or term definitions, based on their own usage of commonly clinical terminology. Dichotomous labels included the presence or absence of RH in each retinal quadrant, type of RH, and presence of other retinal abnormalities. Intergrader agreement was assessed using Cohen's Kappa coefficient.

Results : There was only fair agreement between experts for labels across images from all categories of RH (mean kappa 0.393). This finding was similar when stratified by RH etiology categories (mean kappa 0.375 for medical causes; 0.337 for head trauma, including birth). Presence of optic disc swelling (mean kappa 0.587), cotton wool spots (mean kappa 0.561), and hard exudates (mean kappa 0.498) had good agreement among the experts; however, there was a lack of agreement on the presence of C- or reversed C-shaped hemorrhage along temporal arcade vessels (mean kappa 0.036) across all images.

Conclusions : Dependence upon common clinical definitions alone for labeling of retinal features and RH characteristics when developing a training set for a machine learning algorithm may result in overly variable, and therefore unreliable, image labels due to variability in the interpretation and usage of those terms by ophthalmologists. Developing grader consensus for definitions of terms and features before labeling may reduce variability, ensure accurate labeling, and improve the reliability and validity of expert labeling of RH for machine learning studies.

This abstract was presented at the 2023 ARVO Annual Meeting, held in New Orleans, LA, April 23-27, 2023.