Negative effects of unlabeled response scales

Abstract

This study used a novel research approach to investigate the effects of unlabeled response scales on response distributions. Instead of responding to standard questionnaire items respondents were asked to report given judgments on either semantic-differential (SD) or agree-disagree (AD) response scales, thereby showing the extent to which respondents agree upon where to place given judgments. Results from a survey-based study ( N = 418) show that respondents to a large extent disagree about where to place judgments on the response scale; the level of agreement for different judgment intensities ranged from 42% to 82% and the level of agreement is lower for AD than SD response scales. The low levels of agreement contribute to non-substantive variance in the data which increases the risk of attenuated or inflated correlations between constructs. Moreover, simulations of actual response distributions suggest that unlabeled response scales may lead to a strong bias in the form of underestimated shares of positive answers. Implications for research and marketing research practice of using unlabeled response scales are discussed and it is recommended that response categories on SD and AD items always should be labeled since this will reduce non-substantive variance and bias in the data.