Leveraging Machine Learning to Develop Digital Engagement Phenotypes of Users in a Digital Diabetes Prevention Program: An Evaluation Study (Preprint)

Abstract

Digital diabetes prevention programs (dDPPs) are effective “digital prescriptions” but suffer from high attrition rates and program non-completion. To address this, we developed PAMS, a personalized automatic messaging system that leverages SMS and data integration into clinical workflows to enhance patient-provider communication and increase engagement with a dDPP. Preliminary data showed positive results. Further investigation is needed to evaluate the role of machine learning in developing a more personalized version of PAMS with tailored support technology to boost engagement.

This study leverages machine learning to develop digital engagement phenotypes of dDPP users and assess its accuracy in predicting engagement with dDPP activities. Learning from this research will be used as part of a PAMS optimization process to increase PAMS personalization by incorporating engagement prediction and digital phenotyping. To prove the feasibility of using dDPP user-collected data to build a machine learning model able to predict engagement and contribute to identifying digital engagement phenotypes. To describe methods for developing the machine learning models using our dDPP datasets and present preliminary results. To present preliminary data on user profiling based using the output of our machine learning results.

Using the gradient-boosted forest model, we predicted engagement in four key dDPP individual activities (physical activity, lessons, social activity, and weigh-ins) and general activity (engagement in any activity) based on previous short-term and long-term activity in the application. The area under the ROC curve (AUROC) and the Area under the Precision-Recall curve (AUPRC) metrics determined model performance. Shapley values reflected the feature importance of the models and determined what variables were used for user profiling with latent profile analysis.

We developed two models using weekly and daily DPP datasets (328,821 and 704,242 records, respectively) that yielded predictive accuracies above 90%. Although both models were highly accurate, the daily model better fit our research plan because it predicted daily changes in individual activities, crucial information to be used when creating the “digital phenotypes.” Additionally, to better understand the variables contributing to the model predictor, we calculated the Shapley values for both models to identify the features with the highest contribution to model fit; results suggested that engagement with any activity in the dDPP in the last seven days had the most predictive power. We profiled users with latent profile analysis after two weeks of engagement (BIC = -3222.46) with the dDPP and identified six profiles of users, including those with high engagement, minimal engagement, and attrition.

Preliminary results demonstrate the feasibility of applying ML methods with predicting power as an acceptable mechanism to tailor and optimize messaging interventions to support patient engagement and adherence to digital prescriptions. The results enable future optimization of our existing messaging platform and expansion of this methodology to other clinical domains.

https://www.clinicaltrials.gov/ct2/show/NCT04773834

RR2-10.2196/26750