Rapid evaluation of COVID-19 related service and practice changes in health and human services using tailored methods

Abstract

The COVID-19 pandemic required substantive delivery and practice changes for government services under tight timeframes and high public scrutiny. These urgently implemented service changes provided the opportunity for evaluators to support decision-makers to understand the impact of adaptations for those delivering and receiving health and human services. Tailored rapid evaluation methods (REM) provide a pragmatic approach to generating timely information for evidence-based policy and decision-making under these conditions. Drawing from features of a range of existing rapid evaluation models, as well as developmental and utilization-focussed evaluation theory, this article outlines the design and implementation of a novel REM approach and considers the benefits of both tailoring and standardizing rapid evaluation approaches to meet end-user needs. The tailored REM approach and mixed methods are contextualized and compared to other documented rapid evaluation models to demonstrate the purpose and value of customization. This article builds on previous descriptions of the implementation of a novel REM approach to provide a comparative account of tailored rapid evaluation methods. The article outlines the drivers that led to the selected tailoring of the REM approach, and shares lessons learned in the context of the COVID-19 pandemic by a large internal government evaluation unit (Department of Health and Human Services) in Victoria, Australia. The customized features of REM ensure that it can consider the experiences of those delivering and receiving services, and inform near-term decision-making on programme and policy design in emergency and fast-paced contexts. The article shares a case study of a rapid evaluation of telehealth in pediatric care to demonstrate insights from tailoring the REM approach in practice. The REM method was utilized with the aim of delivering findings in a time-sensitive manner to rapidly inform decision making for policy-makers. Key enablers for the tailored REM protocol include the use of multi-disciplinary teams, flexible evaluation design, and a participatory approach that facilitates stakeholder involvement throughout delivery. Insights from the case study and methods presented seek to inform practice for evaluators who intend to or may want to tailor their own rapid evaluation model in resource and time-limited settings.