Double Entropy Inter-Rater Agreement Indices

Abstract

The proper application of the most frequently used inter-rater agreement indices can be problematic for the case of a single target, for example, a psychotherapy patient, a student’s thesis, a grant proposal, and the lifestyle in a country. The majority of indices that can handle this case assess either the deviation of ranks from some central/average value or the pattern of ranks’ distribution. Contrary to other approaches, this article defines disagreement rating results using the unpredictability/complexity of scores. The article discusses alternative entropy methods for measuring inter-rater agreement or consensus in survey responses for the case of a single target. A new inter-rater agreement index is proposed. Comparisons between this index and the known inter-rater agreement measures show some limitations of the most frequently used indices. Various important methodological issues such as disagreement assumptions, average sensitivity, adjustments to deal with outliers, and missing or incorrectly recorded data are discussed. Examples of applications to actual data are presented.