Predicting Early Dropout in a Digital Tobacco Cessation Intervention: A Replication and Extension (Preprint)

Abstract

Detecting early dropout from digital interventions is crucial for developing strategies to enhance user retention and improve health-related behavioral outcomes. Bricker et al. (2023) proposed a single metric that accurately predicted early dropout from three digital tobacco cessation interventions based on login data in the initial week after registration. Generalization of this method to additional interventions and modalities would strengthen confidence in the approach and facilitate additional research drawing on it to increase user retention.

This study had two research questions (RQ): RQ1) Can Bricker et al. be replicated using data from a largescale observational, multimodal intervention to predict early drop out? RQ2) Can first-week engagement patterns identify users at the greatest risk for early dropout, to inform development of potential “rescue” interventions?

Data were drawn from BecomeAnEX, a freely available, multimodal digital intervention for tobacco cessation. First-week engagement was operationalized as any website pageviews or text message responses within one week post-registration. Early dropout was defined as having no subsequent engagement after that initial week through one year. First, a multivariate regression model was used to predict early dropout. Model predictors were dichotomous measures of engagement in each of the initial 6 days (days 2-7) following registration (day 1). Next, six univariate regression models were compared in terms of their discrimination ability to predict early dropout. The sole predictor of each model was a dichotomous measure of whether users had re-engaged with the intervention by a particular day of the first week (calculated separately for each of days 2-7).

RQ1: The AUC of the multivariate model in predicting dropout after one week was 0.72 (95% CI: 0.71-0.73), which was within the range of AUC metrics found in Bricker et al. RQ2: AUCs of the univariate models increased with each successive day until Day 4 (0.66, 95% CI: 0.65-0.67). Sensitivity of the models decreased (range: 0.79–0.59) and specificity increased (range: 0.48–0.73) with each successive day.

This study provides independent validation of the use of first-week engagement to predict early dropout, demonstrating that the method generalizes across intervention modalities and engagement metrics. As digital intervention researchers continue to address the challenges of low engagement and early dropout, these results suggest that first-week engagement is a useful construct with predictive validity that is robust across interventions and definitions. Future research should explore the utility and efficiency of this model to develop interventions to increase retention and improve health behavioral outcomes.