Some Thoughts about Conservative Evaluations of Replications

Abstract

It is suggested that an unduly conservative research tradition operates in social psychology to heighten the perception of inconsistency in research replications. In large part, this tradition, producing a bias in favor of the null hypothesis, stems from a belief in "the law of small numbers" and a failure to appreciate the probabilistic nature of research results so that each replication or dependent measure is expected to be significant at better than the .05 level by itself: The legacy of R. A. Fisher's approach to inferential statistics, with its emphasis on avoiding Type I errors and neglect of the possibility of Type II errors, is also considered. It is also noted that social psychological tests of hypothesized relationships are typically low in power. he argument in favor of combining probabilities over the series of replications is supported. It may also be, however; that a good many social psychologists are also greatly risk averse, so that the possibility of a negative outcome (making a mistake) is given very much more weight than positive information (favoring the hypothesis being tested by the research).