Not Very Powerful

Abstract

Abstract. Several guidelines on how to construct questionnaire items exist, even though the literature lacks empirical evidence for their effectiveness. To investigate whether the addition of negations and vague quantifiers worsens the psychometric properties of an established questionnaire, 872 participants completed one version of the Positive and Negative Affect Schedule (PANAS) – the German original, a negated version, a version with vague quantifiers or a version with both negations and vague quantifiers. Reliability estimates, item-total correlations, Confirmatory Factor Analysis (CFA) model fit, and fit to the Partial Credit Model (PCM) were compared among the four conditions. No PANAS version was clearly superior as no systematic pattern in the psychometric properties was found. Our findings question the general applicability of the guidelines of item construction as well as the effectiveness of widely used statistical analyses assessing the quality of scales. The results should encourage researchers to put a stronger focus on careful item construction as relying on psychometric properties might not be sufficient to develop valid questionnaires.