Heterogeneity in Meta-Analytic Effect Sizes: An Assessment of the Current State of the Literature

Abstract

Heterogeneity refers to the variability in effect sizes across different samples and is one of the major criteria to judge the importance and advancement of a scientific area. To determine how studies in the organizational sciences address heterogeneity, we conduct two studies. In study 1, we examine how meta-analytic studies conduct heterogeneity assessments and report and interpret the obtained results. To do so, we coded heterogeneity-related information from meta-analytic studies published in five leading journals. We found that most meta-analytic studies report several heterogeneity statistics. At the same time, however, there tends to be a lack of detail and thoroughness in the interpretation of these statistics. In study 2, we review how primary studies report heterogeneity-related results and conclusions from meta-analyses. We found that the quality of the reporting of heterogeneity-related information in primary studies tends to be poor and unrelated to the detail and thoroughness with which meta-analytic studies report and interpret the statistics. Based on our findings, we discuss implications for practice and provide recommendations for how heterogeneity assessments should be conducted and communicated in future research.