Bayesian Test-Retest Repeatability Tells You Want You Want To Know In Equivalency Of Imaging Metrics

Abstract

 

Purpose

 

To investigate Bayesian approaches to assessing test-retest variabilty and compare the findings with standard method comparison statistics.

 

Methods

 

A dataset of objective measures of tarsal conjunctival roughness were used to evaluate the methods. Images of the tarsal conjunctiva (n=29) with various amounts of unevenness in their specular reflection ('roughness'), were manually processed (NIH ImageJ) twice by a technician naive to the experiment’s purpose. Three of the 13 metrics of roughness obtained¹ were evaluated (R_a, R_q & R_t) using a Bayesian approach. Prior distributions of the mean, SD and normality of the difference between the 1st and 2nd evaluation were estimated as wide normal, wide uniform and wide exponential respectively. Posterior distributions were derived from which a joint distribution of effect sizes was generated. The highest density interval (HDI) of the posterior distribution of effect size was compared to a region of practical equivalence (ROPE; Cohen’s d=0.5 & 0.8). The interpretation was compared to standard metrics in method comparison of CCC (R; package epiR) and, mean difference and limits of agreement (LOA).

 

Results

 

Bayes: The mean difference (modal SD) for each parameter were R_a +1.1(1.9), R_q +0.5(2.0) and R_t -0.6(1.7). For a medium effect size (d=0.5) 37%, 87% and 90% of the posterior distribution HDI lay within the ROPE for R_a, R_q and R_t, respectively. The corresponding data for a large effect size (d=0.8) were 86%, 99% and 99%. CCC & LOA: The mean difference (SD) were R_a +1.2(2.0), R_q +0.6(2.5) and R_t -3.32(15.9). Comparative outcomes for R_t are shown in Figure 1.

 

Conclusions

 

In test-retest repeatability, the intention is to characterise the difference between repeat measures. Point estimates of mean difference and LOA may give misleading results when there are departures from concordance. Since the Bayesian approach directly determines the parameter of interest and its varaiblity, acceptance of 'no difference' and gives more insight into equivalency.<br /> ¹Chinga G et al. J Microsc. 2007;227:254-65.  

 

FIgure 1: (Left) )Posterior distributions of the differences, SD of the differences and effect size. The highest density interval (HDI) indicates probable values for each parameter. The region of practical equivalence (ROPE) is shown for Cohen's d=0.8. (Right) Shows evaluation 1 vs. evaluation 2 (top) and the corresponding mean/difference plot (bottom).