Item Response Models, Pathological Science and the Shape of Error

Abstract

There is nothing in Borsboom and Mellenbergh’s (2004) response that refutes my thesis that psychometrics is a pathology of science. They seek to defend item response models from my charge of pathological science without apparently realizing that my charge relates to psychometricians, not to models. They appeal to the Quine-Duhem thesis in an attempt to argue that item response models do not allow the hypothesis that psychological attributes are quantitative to be tested in isolation, but their argument is based upon a misinterpretation of Duhem. In any experiment, what is being tested depends on what the experimenter already takes to be true, and it is possible that a psychometrician could be testing just one of the hypotheses constituting an item response model. Furthermore, using the theory of conjoint measurement, it is possible to isolate predictions that depend upon psychological attributes being quantitative, as opposed to merely ordinal. Despite this, Borsboom and Mellenbergh agree with the first part of my thesis. They do not discuss the second part, but an examination of textbooks on item response models shows that psychometricians disguise their failure to test the hypothesis that psychological attributes are quantitative by simply declining to mention that this hypothesis is presumed in their models. Claims to measure psychological attributes based upon these models depend exclusively upon the weakest part of these models: the hypothesis that the distribution of ‘errors’ takes a specific form.