Leveraging Patient Experience Analysis: Improving the performance of machine learning algorithm in multicentre cohorts (Preprint)

Abstract

In the context of limited resources and the complexities associated with analysing large volumes of the Friends and Family Test (FFT) free-text data, our aim is to create and refine an approach for the deployment of a text analytics algorithm to predict themes from the NHS Patient Experience Framework and sentiment, which can be reliably deployed in healthcare organisations in England.

To enhance the landscape of patient experience nationally, the aim of this study was to refine the previously published algorithm for use in other healthcare settings with acceptable accuracy.

Eleven healthcare organisations with diverse care settings were recruited. Given the variation in care and technical capacity and resource, testing of algorithm across diverse care settings and FFT free-text datasets was performed including manual coding of subset of retrospective comments. Technical infrastructure including coding environment and packages were deployed. The algorithm was tailored to accommodate contextual variations, rectifying issues identified during testing and tested for change in algorithm performance.

The algorithm exhibited satisfactory overall accuracy (>75%) for both themes and sentiment in predicting themes and sentiments embedded within free-text responses. While the classifier yielded strong and reusable models in relatively stable environments, such as adult inpatient care settings, the accuracy was notably lower in organizations providing services such as paediatrics and mental health. The accuracy of our algorithm significantly improved when individual Trust coding templates were applied to these organisations. Thematic saturation was reached after the eighth organisation was recruited and no further coding was required for the last three organisations.

This study represents a significant step forward in leveraging free-text FFT data for valuable insights in healthcare settings through the development of a robust supervised learning text analytics algorithm. The disparity in some care settings was anticipated, given that the lexicon and phraseology used is inherently differ from those prevalent in adult inpatient care (where the algorithm was developed). These challenges were addressed with further coding and testing under various scenarios. This approach also enhanced the accuracy and reliability of the algorithm and encouraged inter- and intra-organisational collaboration and shared-learning.

N/A