THE GHOST IN THE MACHINE? THE VALUE OF EXPERT ADVICE IN THE PRODUCTION OF EVIDENCE-BASED GUIDANCE: A MIXED METHODS STUDY OF THE NICE INTERVENTIONAL PROCEDURES PROGRAMME

Abstract

Objectives: The aim of this study was to determine the aspects of expert advice that decision makers find most useful in the development of evidence-based guidance and to identify the characteristics of experts providing the most useful advice.Methods: First, semi-structured interviews were conducted with seventeen members of the Interventional Procedures Advisory Committee of the UK's National Institute of Health and Care Excellence. Interviews examined the usefulness of expert advice during guidance development. Transcripts were analyzed inductively to identify themes. Second, data were extracted from 211 experts’ questionnaires for forty-one consecutive procedures. Usefulness of advice was scored using an index developed through the qualitative work. Associations between usefulness score and characteristics of the expert advisor were investigated using univariate and multivariate analyses.Results: Expert opinion was seen as a valued complement to empirical evidence, providing context and tacit knowledge unavailable in published literature, but helpful for interpreting it. Interviewees also valued advice on the training and experience required to perform a procedure, on patient selection criteria and the place of a procedure within a clinical management pathway. Limitations of bias in expert opinion were widely acknowledged and skepticism expressed regarding the anecdotal nature of advice on safety or efficacy outcomes. Quantitative analysis demonstrated that the most useful advice was given by clinical experts with direct personal experience of the procedure, particularly research experience.Conclusions: Evidence-based guidance production is often characterized as a rational, pipeline process. This ignores the valuable role that expert opinion plays in guidance development, complementing and supporting the interpretation of empirical data.