Question Wording as an Independent Variable in Survey Analysis

Abstract

This paper renews the line of research into the effects of changes in survey question wording and form which occupied researchers during the 1940s. We suggest two reasons for the cessation of such research: the idiosyncratic nature of many of the items experimented with and the near exclusive focus on single-variable distributions. In the present study, the experiments are based on decisions that face all survey investigators: whether to use agree-disagree statements or forced choice items; whether to ask open or closed questions; whether and how to balance alternatives offered; whether to include a middle alternative; and whether or not to filterfor no opinion. Furthermore, we examine the consequences of these decisions not only for univariate distributions but also for an item's relationship to education. The results from SRC national probability samples suggest thatfor thefirst two types, as well as for items involving variations in tone of word, the decisions may affect inferences about correlations with education. For the other three types the effects are restricted mainly to changes in marginals, although the no-opinion type shows a more limited kind of interaction with education. Finally, we present evidence that index construction is not an adequate solution to the question-wording problem.