Reliability analysis of binary outcomes: sample size and calculations of kappa statistic

Abstract

Reliability analysis is an important methodological tool used in medical research to assess the degree of agreement between measurements taken by different methods or by multiple investigators. In this article, we provide an easy-to-understand overview of the basic concepts associated with reliability analysis, as well as the statistical criteria used in its application in biomedical research. The similarities and differences between the analysis of validity and the analysis of reliability are also presented. The principles of calculating Cohen’s kappa for the simplest situation with two researchers and binary variables are demonstrated both by using the formulas and by applying the SPSS software. Advantages and disadvantages of using kappa statistic are discussed. The article is intended for novice researchers and young scientists and will be useful for planning of research projects and training data collectors.