A look under the hood: analyzing engagement and usage data of a smartphone-based intervention

Abstract

Abstract Background Engagement with smartphone-based interventions stimulates adherence and improves the likelihood of gaining benefits from intervention content. Research often relies on system usage data to capture engagement. However, to what extent usage data reflect engagement is still an open empirical question. We studied how usage data relate to engagement, and how both relate to intervention outcomes. Methods We drew data from a randomized controlled trial (RCT) (N = 86) evaluating a smartphone-based intervention that aims to stimulate future self-identification (i.e., future self vividness, valence, relatedness). General app engagement and feature-specific engagement were retrospectively measured. Usage data (i.e., duration, number of logins, number of days used, exposure to intervention content) were unobtrusively registered. Results Engagement and usage data were not correlated. Multiple linear regression analyses revealed that general app engagement predicted future self vividness (p = .042) and relatedness (p = .004). Furthermore, engagement with several specific features also predicted aspects of future self-identification (p = .005 – .032). For usage data, the number of logins predicted future self vividness (p = .042) and exposure to intervention content predicted future self valence (p = .002). Conclusions Usage data did not reflect engagement and the latter was the better predictor of intervention outcomes. Thus, the relation between usage data and engagement is likely to be intervention-specific and the unqualified use of the former as an indicator of the latter may result in measurement error. We provide recommendations on how to capture engagement and app use in more valid ways.