A 10-year impact assessment of the Efficacy and Mechanism Evaluation (EME) programme: an independent mixed-method evaluation study

Abstract

Background The Efficacy and Mechanism Evaluation (EME) programme – a Medical Research Council (MRC) and National Institute for Health Research (NIHR) partnership – funds trials that evaluate the efficacy of interventions with the potential to promote health and studies that improve our understanding of the mechanisms of underlying diseases and their treatments. Objective To conduct an independent review of the EME programme’s impact and identify opportunities for future improvement. Design A mixed-methods approach, including desk research, an analysis of secondary data, stakeholder consultation and the development of impact case studies. Participants Chief investigators of EME awards, unfunded applicants to the EME programme and key opinion leaders relevant to the programme and research ecosystem. Interventions No interventions were tested, as this was a retrospective programme evaluation. Main outcome measures The evaluation was guided by a set of 15 evaluation questions. Results The EME programme bridges the gap between proof-of-concept and effectiveness studies that are located among other MRC and NIHR schemes and grants from charities in the funding landscape. Mechanistic studies alongside EME trials add value by lending confidence to trial findings and providing insights into the underlying biology. Between 2009 and September 2018, £175.7M in funding was approved for 145 EME projects. EME programme-funded research has started to deliver value to the NHS and patients by improving treatments and providing more efficient use of resources. Of the 43 completed trials, 14% (n = 6) showed that the intervention had a positive effect, whereas 74% (n = 32) of trials did not. The remaining five (12%) trials were unable to recruit participants or did not proceed to the full-trial stage. Seven projects (i.e. 16% of completed trials) have informed clinical guidelines or regulatory approval decisions and another eight projects have the potential to do so in the future, given the nature of their findings. Projects in the EME programme portfolio address a range of UK health needs and government priority areas, but they do not fully align with the level of health needs present. Commissioned calls for applications steer applicants. However, many commissioned calls do not lead to funded awards, and a better understanding of the underlying reasons for this would enable targeted supported to address key health needs. The majority of EME projects investigate existing interventions of limited commercial interest, focusing on repurposing (67/136, 49%) and informing current practice (23/136, 17%). Although there is little evidence of wider economic impact from commercial benefits, the EME programme is important in funding research in which industry is unlikely to invest. Stronger co-ordination with other funders, such as charities, could lead to synergies, enhancing the potential for health impact and influence on other funders’ agendas. The main challenges identified for EME projects were ‘complex and slow contracting processes’ (35/46, 76%), ‘setting up of study sites’ (30/46, 65%) and patient recruitment (28/46, 61%). Enablers of research included a clinical research fellow position on the project and support from Clinical Research Networks and Biomedical Research Centres. Nearly all of the chief investigators consulted had engaged in patient and public involvement at some project stage, and a lack of patient and public involvement did not emerge as a barrier to research or impact. Research ideas stemming from patients were, however, over-represented among unfunded applications, but the reason for this is unclear. Limitations Only about one-third of all studies had been completed or had published their main findings, necessitating a purposive, rather than representative, sampling of the portfolio. The COVID-19 outbreak cut short the programme of interviews, limiting the depth to which some evaluation questions could be explored. Several data sources were based on self-reporting by chief investigators; whereas key self-reported aspects were verified through desk research, this was not possible for all findings. Conclusions The EME programme plays an important role in the UK research funding landscape and has started to deliver value to the NHS and patients. Based on the evidence gathered, seven recommendations were developed to enhance the EME programme’s health and economic impact and address challenges encountered by chief investigators in implementing research projects. Funding This project was funded by the EME programme, a MRC and NIHR partnership. This will be published in full in Efficacy and Mechanism Evaluation; Vol. 8, No. 20. See the NIHR Journals Library website for further project information.