Three Approaches to Improve Inferences Based on Survey Data Collected with Mixed-mode Designs

Abstract

Abstract Mixed-mode designs have become increasingly common in survey data collection. Although different modes often have different measurement properties, the standard practice is to treat mixed-mode data as if they had been collected with a single mode, neglecting the potential impact of mode effects. To account for potential mode effects when making inferences for mixed-mode samples, we propose (i) a Testimator approach, (ii) a Bayesian approach, and (iii) a model averaging approach. In the Testimator approach, we test whether the means and the variances of mixed-mode samples are the same. If the means are the same, we take the average of mode-specific estimates. If the means are different, we take the average when we have no prior information about preferred modes and take the smaller (or larger) estimate when we have prior information about preferred modes (e.g., a smaller estimate is better). In the Bayesian approach, we assume some prior information. We use a data-driven method to determine whether there are mode effects. If there are no mode effects, we draw inferences using a common mean model. If there are mode effects, we draw inferences using the data collected with the mode that produces smaller estimates. In the model averaging approach, we combine estimates of different models (characterized by whether assume same means and variances across modes) using marginal posteriors as weights. We evaluate the approaches in simulation studies and find that they achieve robust inferences compared to the standard approach. We apply the methods to the Arab Barometer study, which employs a randomized mixed-mode design.