Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research

Abstract

Improving the validity and reliability of surveys is a critical part of the response to the call for improved rigor of education research, policy analysis and evaluation. Too often we create inquiry tools without validating our measures against how respondents interpret our questions, and therefore collect data of questionable quality. The purpose of this article is to demonstrate how cognitive interviews can be a useful method for improving the reliability and validity of surveys used in education research. The cognitive interviews described in this article reveal the complex thought processes in which our respondents engaged as they discussed their responses to questions concerning standards, assessments and professional development, and the unanticipated interpretations of seemingly straightforward constructs. We present a number of examples to illustrate complexities that education researchers face in trying to design large-scale surveys to answer critical educational policy questions, and the substantive insights that can be gained from the use of cognitive interviews. The use of cognitive interviews in the development of surveys has been for the most part neglected by the educational research community, and we suggest that more attention be paid to this methodology as a way of improving survey research.