No Evidence that Measuring Moderators Alters Treatment Effects

Abstract

AbstractSocial scientists are frequently interested in who is most responsive to a treatment. By necessity, such moderation experiments often rely on observed moderators, such as partisan identity. These designs have led to an ongoing debate about where to measure moderators—immediately prior to the treatment, after the treatment, or in a prior wave of a panel survey. Measuring a moderator prior to the treatment is the most efficient and avoids posttreatment bias, but it raises concerns about priming. We contribute to this debate by systematically studying whether measuring moderators prior to an experiment affects the results. Across six different experiments, each involving a commonly used moderator, we find little evidence of priming effects, even when a moderator is placed immediately before the experiment. Our findings thus help resolve the debate, suggesting that researchers should measure moderators pretreatment. We conclude with advice on designing well‐powered moderation experiments.