Sources of Variance in the Accuracy of Interviewer Observations

Abstract

In face-to-face surveys, interviewer observations are a cost-effective source of paradata for nonresponse adjustment of survey estimates and responsive survey designs. Unfortunately, recent studies have suggested that the accuracy of these observations can vary substantially among interviewers, even after controlling for household-, area-, and interviewer-level characteristics, limiting their utility. No study has identified sources of this unexplained variance in observation accuracy. Motivated by theoretical expectations from the observer bias literature, this study analyzed more than 45,000 open-ended justifications provided by interviewers in the U.S. National Survey of Family Growth (NSFG) for their observations on two key features of all sampled NSFG households: presence of children and expected probability of household response. The study finds that variability among interviewers in the cues used to record these observations (evident from the open-ended justifications) explains much of the previously unexplained variance in observation accuracy.