Design and validation of a new Healthcare Systems Usability Scale (HSUS) for clinical decision support systems: a mixed-methods approach

Abstract

ObjectiveTo develop and validate a questionnaire to assess the usability of clinical decision support systems (CDSS) and to assist in the early identification of usability issues that may impact patient safety and quality of care.DesignMixed research methods were used to develop and validate the questionnaire. The qualitative study involved scale item development, content and face validity. Pilot testing established construct validity using factor analysis and facilitated estimates for reliability and internal consistency using the Cronbach’s alpha coefficient.SettingTwo hospitals within a single National Health Service Trust.ParticipantsWe recruited a panel of 7 experts in usability and questionnaire writing for health purposes to test content validity; 10 participants to assess face validity and 78 participants for the pilot testing. To be eligible for this last phase, participants needed to be health professionals with at least 3 months experience using the local hospital electronic patient record system.ResultsFeedback from the face and content validity phases contributed to the development and improvement of scale items. The final Healthcare Systems Usability Scale (HSUS) proved quick to complete, easy to understand and was mostly worded by potential users. Exploratory analysis revealed four factors related to patient safety, task execution, alerts or recommendations accuracy, the effects of the system on workflow and ease of system use. These separate into four subscales: patient safety and decision effectiveness (seven items), workflow integration (six items), work effectiveness (five items) and user control (four items). These factors affect the quality of care and clinician’s ability to make informed and timely decisions when using CDSS. The HSUS has a very good reliability with global Cronbach’s alpha 0.914 and between 0.702 and 0.926 for the four subscales.ConclusionThe HSUS is a valid and reliable tool for usability testing of CDSS and early identification of usability issues that may cause medical adverse events.