Logical Versus Empirical Estimates of Item Difficulty

Abstract

The logical estimates made by 5 judges of the difficulty levels of 44 letter series items were compared with the corresponding empirical estimates derived from an actual administration of the items to 186 undergraduates. The results indicated (a) substantial correspondence ( P < .01) between logical and empirical estimates and (b) a great deal of variation among judges in the relative accuracy of their judgments. Written reports of the judges of the rules they used in determining item difficulty indicated that the more elaborate a judge was in analyzing the structure and organization of an item, the more accurate was his estimation of the item difficulty level.