Detecting Relative Bias Using the Bland-Altman Method: Consequences of Unequal Imprecision

Abstract

Purpose: : Bland-Altman plots are often used to determine the relative bias between two methods of measurement. This approach assumes that the two methods have equal imprecision but this strong assumption is often unchecked. This study demonstrates the consequences when imprecisions are unequal.

Methods: : Using simulated intraocular pressure (IOP) data for two devices (N=1000) four models were constructed to illustrate possible Bland-Altman scenarios encountered by researchers. In the first model both methods have equal imprecision standard deviations (SD) equal to 3 mmHg and a bias between the devices increasing 0.4 mmHg per each 1 mmHg increase in the true level. The second had equal imprecision (same as Model 1) and no bias. The third had unequal imprecisions (SDs of 2.4 and 3 mmHG) and no bias. The fourth had unequal imprecision (SDs of 3.69 and 3 mmHg) and the same bias as the first model.

Results: : In the two models that held to the assumption of equal imprecision (Figure 1A and 1B), the Bland-Altman method correctly identified the bias (P<0.0001) in the first model and the lack of bias (P=0.6534) in the second model. In the third model (Figure 1C), Bland-Altman incorrectly concluded that there was a bias (P<0.0001). In the fourth model (Figure 1D), Bland-Altman failed to detect the bias (P=0.7609).

Conclusions: : In cases where two devices have the same imprecision, Bland-Altman plots are effective tools for determining relative bias. When this assumption is violated, both false positives and false negatives can occur that are indistinguishable from true cases. Therefore, at minimum researchers must strongly justify the assumption of equal imprecision when using Bland-Altman with only two methods due to the high risk of erroneous conclusions. When this is not feasible, researchers should follow Bland and Altman's recommendation to repeat both measurements in order to allow simultaneous estimates of bias and imprecision which avoids making a possibly unrealistic assumption of equal imprecision.

Keywords: clinical (human) or epidemiologic studies: biostatistics/epidemiology methodology • intraocular pressure