Effectiveness of a multilingual digital mental health and wellbeing chatbot (ChatPal): a mixed-methods and multi-centre trial (Preprint)

Abstract

In recent years, advances in technology have led to an influx of mental health apps, in particular the development of mental health and wellbeing chatbots which have already shown promise in terms of their efficacy, availability and accessibility. The ChatPal chatbot was developed to promote positive mental wellbeing amongst citizens living in rural areas. ChatPal is a multilingual chatbot, available in English, Scottish Gaelic, Swedish and Finnish, containing psychoeducational content and exercises such as mindfulness and breathing, mood logging, gratitude and thoughts diaries.

The primary objective of this study is to establish if using a multilingual mental health and wellbeing chatbot (ChatPal) has an effect on mental wellbeing. Secondary objectives include characterising individuals whose wellbeing improved or declined, and applying thematic analysis to the user feedback.

Participants were recruited to use the chatbot over a 12-week trial period. ChatPal was trialled in five regions; Northern Ireland, Scotland, Republic of Ireland, Sweden and Finland. Outcome measures included the Short Warwick Edinburgh Mental Wellbeing Scale, World Health Organisation-Five Well-Being Index, and Satisfaction with Life Scale, which were recorded at baseline, midpoint and end-point. Written feedback was collected from participants throughout the trial and subjected to qualitative analysis to identify themes.

A total of 348 people were recruited to the trial (75% female, 25% male) aged between 18-73 (mean age 30). The wellbeing scores of participants improved from baseline to midpoint and from baseline to end-point, however improvement in scores were not statistically significant on the Short Warwick Edinburgh Mental Wellbeing Scale (P=.42), World Health Organisation-Five Well-Being Index (P=.52), or Satisfaction with Life Scale (P=.81). Individuals that did show improvements in wellbeing (n=16) interacted more with the chatbot and were significantly younger compared (average 27 years old) to those whose wellbeing declined over the trial. Three themes were identified from user feedback including ‘positive experiences’, ‘mixed/ neutral experiences’ and ‘negative experiences’. Positive experiences mentioned enjoying exercises provided by the app, while most of the mixed/ neutral or negative experiences mentioned liking the chatbot overall, but that there were some barriers, such as technical or performance errors which needed to be overcome.

This paper further supports the need for blended service offerings. Marginal improvements in mental wellbeing were seen in those who used ChatPal (albeit non significant), so we propose that the chatbot could be used along with different service offerings to complement other digital or face-to-face services.