The Impact of Purpose On Rating Quality and User Acceptance

Abstract

Using a quasi-experimental design, the effects of purpose (evaluative vs. developmental) on both peer-rating quality and user acceptance were examined. Subjects were 65 undergraduates divided into 11 project groups. Six groups conducted peer ratings for evaluative (i.e., grading) purposes, whereas the remaining 5 did so for the purpose of providing developmental feedback. Peer ratings conducted for evaluative purposes tended to contain greater halo and to be more lenient, less differentiating, less reliable, and less valid than those performed for developmental purposes. User acceptance as measured by recommendation for future use was more favorable under the developmental than the evaluative conditions. These results suggest that the quality of peer ratings and user acceptance are highly susceptible to the influence of rating contexts and that peer ratings are more useful for developmental than for evaluative purposes. Implications of these results for future peer-appraisal practices and research are discussed.