What the Average Really Means: Dissociating Effect Size and Effect Prevalence usingp-curve Mixtures

Abstract

AbstractMost research in the behavioral sciences aims to characterize effects of interest using sample means intended to describe the “typical” person. A difference in means is usually construed as a size difference in an effect common across subjects. However, mean effect size varies with bothwithin-subject effect sizeandpopulation prevalence(proportion of population showing the effect) in compared groups or across conditions. Few studies consider how prevalence affects mean effect size measurements and existing estimators of prevalence are, conversely, confounded by uncertainty about within-subject power. We introduce a widely applicable Bayesian method, thep-curve mixture model, that jointly estimates prevalence and effect size. Our approach outperforms existing prevalence estimation methods when within-subject power is uncertain and is sensitive to differences in prevalence or effect size across groups or experimental conditions. We present examples, extracting novel insights from existing datasets, and provide a user-facing software tool.