Advantages and challenges in using mobile apps for field experiments: A systematic review and a case study

Abstract

As smartphone’s computing power continues to grow and as mobile applications (apps) continue to dominate digital engagement, apps have become a new frontier for advancing field experiment methodology. Using apps may help researchers to scale up the reach, precisely control randomization and experiment materials, collect a variety of objective and self-reported data over time, and more conveniently replicate and adapt an experiment. We performed a systematic review on field experiments involving apps published between 2007 and 2017. Seven databases were scanned using a predefined search strategy. The database search retrieved 4,810 citations; 101 articles met the inclusion criteria. Our review suggests that scholars have only started to employ apps in field experiments in the last 4 years. Most studies only used apps as an experiment treatment instead of an experiment platform; therefore, researchers have yet to fully leverage the advantages. Almost all studies were from the health research domain and 77.2% used randomized controlled trial design. Only 7 studies utilized smartphone sensors for collecting data. Only one study reported cost and ethical concerns regarding using apps for the experiment. Given these findings, we reported a case study that targeted a minority racial group and leveraged the advantages of apps as an experiment platform and as a data collection tool to illustrate practical challenges and lessons learned regarding time, financial cost, and technical support. In conclusion, we suggest apps provide new ways to study causal mechanisms with experiment big data. Limitations of generalizability, retention, and design quality were discussed as well.