A systematic review of factors affecting the judgments produced by formal consensus development methods in health care

Abstract

Objectives: Formal consensus development methods are ways of obtaining and synthesising views of experts, opinion leaders and other stakeholders, and are increasingly being used to develop clinical practice guidelines. Our objective was to examine the impact that the characteristics of individual participants, groups and the consensus process have on the judgments produced by formal consensus development methods in health care. Methods: Studies were identified from an earlier methodological review and a search of five bibliographic databases for the period January 1996 to December 2004. Studies were eligible if they involved formal consensus development methods and reported differences in judgments between groups or participants. For studies comparing two or more groups overall percentage agreement, the kappa coefficient and the odds ratio for differences in judgments were calculated. Results: There were 22 studies comparing the impact of the characteristics of individual participants within groups and 30 studies comparing the results produced by two or more groups. Practitioners who perform a procedure tend to emphasise the appropriateness of the procedure compared with non-performing practitioners, and individuals from groups that were subject to performance criteria are more critical of those criteria than individuals from other groups. There was no clear pattern for the differences in judgments produced by participants and groups from different countries. Conclusions: Except for participant specialty there is little generalisable evidence for how the characteristics of participants and groups influence the judgments produced in formal consensus development methods. Multi-specialty groups are preferable to single-specialty groups because of their potential for taking account of a wider range of opinions.