A general approach to evaluating agreement between two observers or methods of measurement from quantitative data with replicated measurements

Abstract

We present a general approach to the definition and estimation of coefficients for evaluating agreement between two fixed methods of measurements or human observers. The measured variable is assumed to be continuous with a finite second moment. No other distributional assumptions are made. We introduce the term `disagreement function' for the function of the observations that is used to quantify the extent of disagreement between the two measurements made on the same subject. The proposed inter-methods agreement coefficients compare the disagreement between measurements made by different methods on the same subject to the corresponding disagreement between replicated measurements made by the same method. Therefore, the new coefficients require data with replications readings. We propose inter-methods agreement coefficients for two practical situations involving two methods that have a measurement error: 1) comparison of a new method to a gold standard (or a reference method), and 2) comparison of two methods where neither method is considered a gold standard. We consider three disagreement functions based on the differences between two measurements: 1 ) the mean squared difference, 2) the mean absolute difference and 3 ) the mean relative difference. We then derive non-parametric estimates for the various agreement coefficients. Our approach is illustrated using data from a study comparing systolic blood pressure measurements by a human observer and an automatic monitor. The performance of the new estimates is assessed via stochastic simulations.