Parameter Estimation for Peer Grading under Incomplete Design

Abstract

Methodology developed in the paper supports administration and analysis of peer grading exercises. The technique is especially useful for the incomplete design case, wherein each grader does not grade each paper. The procedure produces simultaneous estimates of (a) bias of each grader, (b) "true" grade for each paper, and (c) relative consistency of each grader. The approach is based on a simple linear statistical model, augmented by a zerosum constraint on grader bias parameters. Ordinary least squares (OLS) estimates are computed and then refined via generalized least squares (GLS) estimation. Case weights for the GLS are based on consistency patterns of the graders, as revealed in the OLS step. The methodology is applied to artificial data to demonstrate its effectiveness. The methodology applies to any situation wherein multiple raters rate multiple ratees on an additive scale.