Relationship between Order of Variables and Determination of Policies When Using Judgment Analysis

Abstract

The methodology for conducting a Judgment Analysis requires judges to rate or rank profiles comprised of a number of variables representing several stimulus dimensions. In this study we investigated the relationship between the order of variables (fixed or random) and 19 judges' rating policies. The results of the Judgment Analysis for the two methods of sequencing profile variables indicate that there is an association between the order of variables and the judges' ability to utilize them in their rankings. When variables were placed in the same order on all profiles, judges used more variables in creating their policies than when the order of the variables randomly changed from profile to profile.