Evolution of a multilevel framework for health program evaluation

Abstract

A well-conceived evaluation framework increases understanding of a program’s goals and objectives, facilitates the identification of outcomes and can be used as a planning tool during program development. Herein we describe the origins and development of an evaluation framework that recognises that implementation is influenced by the setting in which it takes place, the individuals involved and the processes by which implementation is accomplished. The framework includes an evaluation hierarchy that focuses on outcomes for consumers, providers and the care delivery system, and is structured according to six domains: program delivery, impact, sustainability, capacity building, generalisability and dissemination. These components of the evaluation framework fit into a matrix structure, and cells within the matrix are supported by relevant evaluation tools. The development of the framework has been influenced by feedback from various stakeholders, existing knowledge of the evaluators and the literature on health promotion and implementation science. Over the years, the framework has matured and is generic enough to be useful in a wide variety of circumstances, yet specific enough to focus data collection, data analysis and the presentation of findings. What is known about the topic? Evaluation in healthcare typically investigates the implementation of complex innovations in uncontrolled ‘real world’ settings. This presents many challenges for evaluators and those wanting to commission evaluations. What does this paper add? Herein we describe the origins and development of an evaluation framework that is structured in terms of three levels and six domains. The framework has stood the test of time and been used to evaluate a variety of innovations in the delivery of health and aged care. Novel aspects of the framework include the concepts of levels, capacity building and sustainability. What are the implications for practitioners? The structure of the evaluation framework is simple, yet comprehensive, and is suitable for use in a wide range of settings. The framework can also be used to both evaluate and plan the implementation of innovations in health service delivery.