Controversies in meta-analysis: the case of the trials of serum cholesterol reduction

Abstract

There has recently been disagreement in the literature on the results and interpretation of meta-analyses of the trials of serum cholesterol reduction, both in terms of the quantification of the effect on ischaemic heart disease and as regards the evidence of any adverse effect on other causes of death. This paper describes statistical aspects of a recent meta-analysis of these trials, and draws some more general conclusions about the methods used in meta-analysis. Tests of an overall null hypothesis are shown to have a basis clearly distinct from the more extensive assumptions needed to provide an overall estimate of effect. The fixed effect approach to estimation relies on the implausible assumption of homogeneity of treatment effects across the trials, and is therefore likely to yield confidence intervals which are too narrow and conclusions which are too dogmatic. However the conventional random effects method relies on its own set of unrealistic assumptions, and cannot be regarded as a robust solution to the problem of statistical heterogeneity. The random effects method is more usefully regarded as a type of sensitivity analysis in which the weights allocated to each study in estimating the overall effect are modified. However, rather than using a statistical model for the 'unexplained' heterogeneity, greater insight and scientific understanding of the results of a set of trials may be obtained by a careful exploration of potential sources of heterogeneity. In the context of the cholesterol trials, the heterogeneity according to the extent and duration of cholesterol reduction are of prime concern and are investigated using logistic regression. It is concluded that the long-term benefits of serum cholesterol reduction on the risk of heart disease have been seriously underestimated in some previous meta-analyses, while the evidence for adverse effects on other causes of death have been misleadingly exaggerated.