Invalidity of True Experiments

Abstract

The validity of true experiments is threatened by a class of self-report biases that affect all respondents at pretest but which are diminished by treatment, yielding noncomparable treated and control subjects at posttest. These biases include inaccurate self-evaluations due to (a) lack of understanding of dimensions of self-rating, (b) unconscious needs to rationalize sources of severe emotional distress, (c) distressed or altered states (e.g., drug states) that lower the ability to report accurately, and (d) deliberate impression management in service of accessing desired treatment. They are detectable via external criteria, special conditions of measurement, and retrospective pretests, and may be lessened in several ways. Unchecked, they produce treatment by self-report bias interactions that undermine the validity of even true experiments.