Analysis of agreement of clinical experts to B-scan of interest algorithm

Abstract

Purpose : Optical coherence tomography (OCT) scans are important for clinical evaluation of the retina, but manual evaluation of each B-scan is time consuming and requires significant expertise. To improve the workflow, a B-scan of interest tool (BSOI) was created using deep learning to indicate B-scans of interest as well as B-scans of poor image quality (IQ). The purpose of this study was to determine if the BSOI tool agreed with doctors’ clinical judgement.

Methods : Macular cube 512x128 scans were acquired using CIRRUS™ 6000 (ZEISS, Dublin, CA) on 47 subjects with various pathologies including macular degeneration, diabetic eye disease, epiretinal membranes, macular hole and central serous retinopathy. The BSOI tool first runs an IQ algorithm which flags B-scans with poor IQ then further flags good IQ scans if it finds one of the retinal pathologies as referenced in [1]. The prototype tool was integrated into the macular thickness analysis report, flagging B-scans of interest with a red indicator and poor IQ scans with a yellow indicator.
5 independent expert graders (2 retina specialists (RS) and 3 optometrists (OD)) evaluated the tool results and used a feedback feature to indicate if they agreed (thumbs up) or disagreed (thumbs down) (Fig 1). Intraclass correlation coefficient (ICC) and pair-wise Cohen’s kappa between graders were used to assess inter-grader consensus. Graders also answered a survey rating the clinical utility and loading time of the tool.

Results : Mean agreement of an individual grader with the tool was 79% and majority vote agreement between graders and the tool was 81%. Table 1 shows agreement (and 95% confidence interval) of each grader and survey answers. The ICC was 0.44 (0.32, 0.60) indicating poor reliability of agreement between all doctors. Kappas showed fair to moderate agreement between pairwise doctors.

Conclusions : This tool successfully flags B-scans that may have otherwise been missed by doctors. When evaluating agreement between graders on individual cases, differences in inter-clinician's grades were observed, even between retinal specialists. The mean agreement between the B-scan of interest algorithm and the graders were considered clinically acceptable. All graders agreed that the tool increases efficiency with appropriate loading time, can be useful for diagnoses and can be used by both technicians as well as clinicians.
References:
[1] Yu et al. IOVS 2020; 61(9): Abstract PB0085.

This is a 2021 ARVO Annual Meeting abstract.