Rater Agreement Indexes for Performance Assessment

Abstract

Under the heading of reliability, most textbooks refer to classical reliability indexes as appropriate measures for determining interrater agreement. It is argued that interrater agreement is a psychometric property that is theoretically different from classical reliability. Interrater agreement indexes measure the degree to which two or more raters agree on the observation of one or more behaviors on one or more subjects and are not theoretically related to classical test theory. A detailed set of formulas is presented to illustrate a set of algebraically equivalent rater agreement indexes that are intended to provide the educational and psychological researcher and practitioner with a practical means of establishing a measure of rater agreement. The formulas are illustrated with a data set. The formulas can be used for dichotomous and continuous data for two or more raters, on one or more subjects, on one or more behaviors. These rater agreement indexes are useful with performance assessments such as observations, porfolios, performance evaluations, essay writing evaluations, authentic assessments, and so on, where multiple facets impact rater agreement.