Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide

Abstract

The authors present guidelines for establishing a useful range for interrater agreement and a cutoff for acceptable interrater agreement when using Burke, Finkelstein, and Dusig’s average deviation (AD) index as well as critical values for tests of statistical significance with the AD index. Under the assumption that judges respond randomly to an item or set of items in a measure, the authors show that a criterion for acceptable interrater agreement or practical significance when using the AD index can be approximated as c/6, where c is the number of response options for a Likert-type item. The resulting values of 0.8, 1.2, 1.5, and 1.8 are discussed as standards for acceptable interrater agreement when using the AD index with 5-, 7-, 9-, and 11-point items, respectively. Using similar logic, the AD agreement index and interpretive standard are generalized to the case of a response scale that involves percentages or proportions, rather than discrete categories, or at the other extreme, the assessment of interrater agreement with respect to the rating of a single target on a dichotomous item (e.g., yes-no, agree-disagree, true-false item formats). Finally, the usefulness of these guidelines for judging acceptable levels of interrater agreement with respect to the metric (or units) of the original response scale is discussed.