The Use of Multimode Data Collection in Random Digit Dialing Cell Phone Surveys for Young Adults: Feasibility Study

Abstract

Background Young adults’ early adoption of new cell phone technologies have created challenges to survey recruitment but offer opportunities to combine random digit dialing (RDD) sampling with web mode data collection. The National Young Adult Health Survey was designed to test the feasibility of this methodology. Objective In this study, we compared response rates across the telephone mode and web mode, assessed sample representativeness, examined design effects (DEFFs), and compared cigarette smoking prevalence to a gold standard national survey. Methods We conducted a survey experiment where the sampling frame was randomized to single-mode telephone interviews, telephone-to-web sequential mixed mode, and single-mode web survey. A total of 831 respondents aged 18 to 34 years were recruited via RDD at baseline. A soft launch was conducted prior to main launch. We compared the web mode to the telephone modes (ie, single-mode and mixed mode) at wave 1 based on the American Association for Public Opinion Research response rate 3 for screening and extended surveys. Base-weighted demographic distributions were compared to the American Community Survey. The sample was calibrated to the US Census Bureau's American Community Survey to calculate DEFFs and to compare cigarette smoking prevalence to the National Health Interview Survey. Prevalence estimates are estimated with sampling weights and are presented with unweighted sample sizes. Consistency of estimates was judged by 95% CI. Results The American Association for Public Opinion Research response rate 3 was higher in the telephone mode than in the web mode (24% and 30% vs 6.1% and 12.5%, for soft launch and main launch, respectively), which was reflected in response rate 3 for screening and extended surveys. During the soft launch, the extended survey and eligibility rate were low for respondents pushed to the web mode. To boost productivity and survey completes for the web condition, the main launch used cell phone numbers from the sampling frame where the sample vendor matched the number to auxiliary data, which suggested that the number likely belonged to an adult in the target age range. This increased the eligibility rate, but the screener response rate was lower. Compared to population distribution from the US Census Bureau, the telephone mode overrepresented men (57.1% [unweighted n=412] vs 50.9%) and those enrolled in college (40.3% [unweighted n=269] vs 23.8%); it also underrepresented those with a Bachelor of Arts or Science (34.4% [unweighted n=239] vs 55%). The web mode overrepresented White, non-Latinos (70.7% [unweighted n=90] vs 54.4%) and those with some college education (30.4% [unweighted n=40] vs 7.6%); it also underrepresented Latinos (13.6% [unweighted n=20] vs 20.7%) and those with a high school or General Education Development diploma (15.3% [unweighted n=20] vs 29.3%). The DEFF measure was 1.28 (subpopulation range 0.96-1.93). The National Young Adult Health Survey cigarette smoking prevalence was consistent with the National Health Interview Survey overall (15%, CI 12.4%-18% [unweighted 149/831] vs 13.5%, CI 12.3%-14.7% [unweighted 823/5552]), with notable deviation among 18- to 24-year-olds (15.6%, CI 11.3%-22.2% [unweighted 51/337] vs 8.7%, CI 7.1%-10.6% [unweighted 167/1647]), and those with education levels lower than Bachelor of Arts or Science (24%, CI 19.3%-29.4% [unweighted 123/524] vs 17.1%, CI 15.6%-18.7% [unweighted 690/3493]). Conclusions RDD sampling for a web survey is not feasible for young adults due to its low response rate. However, combining this methodology with RDD telephone surveys may have a great potential for including media and collecting autophotographic data in population surveys.