A method to the madness? Quantitative research reviewing

Abstract

Quantitative research reviewing, or meta-analysis, is increasingly popular in the USA, but has made little impact in Britain. Proponents of this technique claim that it can overcome the subjectivity inherent in traditional research reviews and synthesise large numbers of findings in a way which reveals their underlying trends. This paper discusses the claims and criticisms of meta-analysis. It suggests that quantitative reviewing is not a supplement to narrative reviewing, but fulfills rather different functions. It may also be more suited to the climate of educational research in America than in Britain. Nevertheless it is an important development which British researchers should not ignore. A new technique is transforming educational research in America — and incidentally generating a new optimism both in the power of educational research and in the effectiveness of schools to bring about change. The technique is metaanalysis. Essentially, it is a quantitative method of reviewing and summarising the findings of research on any particular topic. It started in the late 1970's, and a few ripples crossed the Atlantic with reports of a synthesis of research on the relationship between class size and achievement. This confirmed what teachers had always maintained but educational researchers had disputed — that better results were obtained with smaller classes (Glass and Smith, 1979; Smith and Glass, 1980). Since then, little has been heard of meta-analysis in this country, but it has expanded rapidly in the USA. Virtually every topic of established interest has been reviewed in this way, ranging from the efficacy of new science curricula (Shyman-sky et al., 1983) to that of open education (Giaconia and Hedges, 1977), and from the effect of socio-economic status (SES) on achievement (White, 1982) to that of graded homework (Paschal et al., 1985). Part of the reason for this surge of interest is that in general the quantitative reviewers conclude that the results are larger and less contradictory than was thought on the basis of the traditional narrative review. Based largely on the results of these meta-analyses, writers such as Bloom (1984) and Walberg (1984) have developed theories of educational productivity in which they argue that alterable variables, whether in schools or in homes, are the major component of school achievement. This viewpoint contrasts strongly with the socio-economic determinism which was dominant throughout the 1970's. I have recently become acquainted with meta-analysis, both through reading completed reviews and through undertaking to write one myself. In this paper I will examine the rationale underlying this development in educational research and assess how the theory stands up in practice. Some of the advantages and disadvantages of the approach are discussed, with particular reference to the possibility of quantitative reviews becoming as widespread in Britain as they are in the USA. I will not attempt to describe the methodology in any detail as a number of textbooks are now available (Glass et al., 1981; Cooper, 1984; Rosenthal, 1984; Hedges and Olkin, 1985). Fitzgibbon (1984) has recently provided an introduction for British readers.