Replicability and generalizability in population psychiatric neuroimaging

Abstract

AbstractStudies linking mental health with brain function in cross-sectional population-based association studies have historically relied on small, underpowered samples. Given the small effect sizes typical of such brain-wide associations, studies require samples into the thousands to achieve the statistical power necessary for replicability. Here, we detail how small sample sizes have hampered replicability and provide sample size targets given established association strength benchmarks. Critically, while replicability will improve with larger samples, it is not guaranteed that observed effects will meaningfully apply to target populations of interest (i.e., be generalizable). We discuss important considerations related to generalizability in psychiatric neuroimaging and provide an example of generalizability failure due to “shortcut learning” in brain-based predictions of mental health phenotypes. Shortcut learning is a phenomenon whereby machine learning models learn an association between the brain and an unmeasured construct (the shortcut), rather than the intended target of mental health. Given the complex nature of brain-behavior interactions, the future of epidemiological approaches to brain-based studies of mental health will require large, diverse samples with comprehensive assessment.