Usability of an Automated System for Real-time Monitoring of Shared Decision Making for Surgery: A Mixed Methods Evaluation (Preprint)

Abstract

Improving shared decision making (SDM) for patients has become a health policy priority in many countries. Achieving high-quality SDM is particularly important for surgical treatment decisions. Large-scale monitoring of surgical patients’ experience of SDM in real-time is needed to identify failings of SDM before surgery happens. We developed a novel approach to automating real-time data collection through an electronic measurement system to address this. Examining usability will facilitate its optimization and wider implementation to inform interventions aiming to improve SDM.

This study examined the usability of an electronic, real-time measurement system to monitor surgical patients’ experience of SDM. Objectives were to evaluate metrics and indicators relevant to system effectiveness, system efficiency and user satisfaction.

A mixed-method usability evaluation used multiple participant cohorts. The measurement system was implemented in a large acute care hospital providing metrics for data collected between April and December 2021. Patients’ experience of SDM was assessed prior to surgery using 2 validated measures (CollaboRATE, SDM-Q-9) sent via email or SMS text messaging. The study invited adult patients booked for urgent and elective surgery across 7 specialties. Patients without capacity to consent for medical procedures, those without access to an appropriate internet-enabled device and those undergoing emergency and endoscopic procedures were excluded. User testing sessions and semi-structured interviews with 2 groups of service users were performed. Participants were individuals from 2 patient experience panels who had not completed the measurement system (Group 1), and a sub-set of patients who had completed the measurement system (Group 2). Quantitative data analysis used descriptive statistics. Calculated were task completion rate and survey response rate (system effectiveness) and task completion time, task efficiency, and relative efficiency (system efficiency). Qualitative thematic analysis identified indicators of good usability (user satisfaction).

A total of 5,794 surgical patients received invitations to complete the measurement. Some 25 service users (n=9 in Group 1; n=16 in Group 2) participated in user testing sessions and interviews. Task completion was high (169/171, 99%) and survey response rate was good (2,254, 39%). Median task completion time was 3 minutes (IQR:2,13), suggesting good system efficiency and effectiveness. The measurement system was perceived as acceptable, easy to access and easy to use, indicating good user satisfaction. Service users identified potential barriers and solutions to acceptability and ease of access that can inform strategies for optimizing the measurement system in future.

A mixed-method evaluation of an electronic measurement system for automated, real-time monitoring of patients’ experience of SDM showed that usability amongst patients was high. Findings will be used alongside further evaluation to demonstrate scalability of the system. Future pilot work will optimize the system for wider implementation to ultimately inform intervention development to improve SDM.