Intervention use during week one allows for machine learning based prediction of adherence in the iFightDepression tool – Analysis of routine care log-data of an online intervention for depression (Preprint)

Abstract

Online interventions, such as the iFightDepression tool (iFD tool) are increasingly being used as an efficacious alternative to classical face-to-face psychotherapy or pharmacotherapy. But especially when used outside of study settings, low adherence rates and the resulting reduced benefit of the intervention limit their effectiveness. Knowledge of factors predicting adherence would enable early, tailored interventions for those at risk for non-adherence.

To identify users at risk for processing the online intervention only incompletely or not at all, this study aimed to predict adherence of using the iFD tool from characteristics obtained during baseline and week one of the intervention use in patients with depression.

Log-data of N=4365 adult patients who have registered to the iFD tool between October 2016 and May 2022 and gave informed consent to participate in the ongoing evaluation were statistically analysed. The resulting dataset was divided into a test (30%) and a training data set (70%) at random. The training data set was then used to train a random forest model to predict the adherence of each user at the start of the intervention using the hypothesized predictors (age, self-reported sex, expectations of the intervention, current or preceding depression treatments, confirmed diagnosis of depression, PHQ-9 score at the beginning of the intervention and usage behaviour within the first week). After training the random forest model, it was fed with the actual test data set to verify if the prediction of adherence by our algorithm was accurate and precise.

Of all patients evaluated, 1170 (27%) were considered adherent according to our predefined definition. A random forest model based solely on sociodemographic and clinical predictors obtained at baseline did not allow for a statistically significant prediction of adherence. After including the usage behaviour of each patient within the first week of the intervention, a significant prediction of adherence was achieved (p < .001). Within this prediction the random forest achieved an accuracy of 0.81 (95% CI: 0.79-0.83), a F1-score of 0.58 and an AUC of 0.84 as well as a specificity of 0.93 for predicting non-adherent users.

Our results demonstrate that the usage behaviour in the first week of the online intervention has a far greater impact as a predictor for adherence than any sociodemographic or clinical factors. Therefore, examining the usage behaviour within the first week and identifying non-adherers through the algorithm could be beneficial to tailor interventions for improving user adherence in a targeted manner e.g. through telephone calls or automated reminder emails with optimal resource utilization.

https://aspredicted.org/4hh34.pdf