Assessing and quantifying inter-rater variation for dichotomous ratings using a Rasch model

Abstract

We present a new model-based approach to the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. The model is a logistic regression model with random effects – a Rasch model. In the rater setting, the Rasch model includes parameters that allow raters to have different propensities to score a given set of individuals positively or negatively – the rater bias. An exact score test of the hypothesis of no rater bias is proposed and is shown to be an exact generalised McNemar's test. Based on the model, we suggest quantifying the rater variation as a suitable measure of the variation of the rater odds ratios. An important example that will serve to motivate and illustrate the proposed model, is the study of Umbilical artery Doppler velocimetry used by obstetricians to assess the status of a foetus. The purpose of the assessment is to improve the foetus' chance of survival by choosing the optimal time of elective delivery. In the study, data related to 139 perinatal deaths were sent to 32 experts who were asked whether the use of Doppler velocimetry might have prevented each death.