Assessing the performance of health technology assessment (HTA) agencies: developing a multi-country, multi-stakeholder, and multi-dimensional framework to explore mechanisms of impact

Abstract

Abstract Background Health technology assessment (HTA) agencies have an important role to play in managing the rising demands on health systems. However, creating and running such agencies potentially diverts resources from frontline services. A large number of studies address the question of ‘what is the impact of HTA?’. Several points of heterogeneity in this literature include: purpose of the study, definition of HTA, definition of impact, and scope and rigour of evaluations. Our study seeks to address several limitations in this literature. This study aims to explore the mechanisms of impact of an HTA agency. In doing so, we consider HTA as an institution rather than a knowledge product to build an impact evaluation framework from an international, multi-stakeholder and multi-dimensional perspective. Methods We conducted 9 key informant interviews with experts from the international HTA community. We addressed several questions, informed by existing frameworks of impact within the literature, to understand their perspectives on the mechanisms of impact of an HTA agency. We analyse data using logic modelling and impact mapping, as tools to understand and visualise mechanisms of change. Findings Our impact mapping highlights several distinct, but not necessarily mutually exclusive, mechanisms through which the overall impact of an HTA agency is achieved. These are: the effective conduct of HTA studies; effective use of HTA in agenda-setting and policy formulation processes; effective engagement and external communications; good institutional reputation and fit within the healthcare and policy-making system; effective use of HTA as a tool for the negotiation of health technology prices; and the effective implementation of policy change regarding health technologies. We also identify indicators of these effects. Conclusions Our findings and resulting evaluation framework complement and add to existing literature by offering a new perspective on the mechanisms by which HTA agencies generate impact. This new perspective considers HTA as an institution rather than a knowledge product, is international, multi-dimensional, and includes multi-stakeholder views. We hope the analysis will be useful to countries interested in managing HTA performance.