Data quality and response distributions in a mixed-mode survey

Abstract

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition. Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode. We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.