A Generalized Kappa Coefficient

Abstract

Previously proposed methods for calculating the kappa measure of nominal rating agreement among multiple raters are not applicable in many situations. This paper presents a more general computational method which can be used across a broader range of rating designs, including those in which raters vary with respect to their base rates and pairs of raters vary with respect to how many subjects they rate in common. A Monte Carlo method for determining the statistical significance of this generalized kappa coefficient is discussed.