Data-driven estimates of the reproducibility of univariate BWAS are biased

Abstract

AbstractRecently, Marek, Tervo-Clemmenset al.1leveraged consortium neuroimaging data to answer a question on most researchers’ minds: how many subjects are required for reproducible brain-wide association studies (BWAS)? Their approach could be considered a framework for testing the reproducibility of several neuroimaging models and measures. Here we test part of this framework, namely estimates of statistical errors of univariate brain-behaviour associations obtained from resampling large datasets with replacement. We suggest that reported estimates of statistical errors are largely a consequence of bias introduced by random effects when sampling with replacement close to the full sample size. We show that these biases can be largely avoided by only resampling up to 10% of the full sample size. Using this unbiased approach, sample size requirements for reproducible univariate BWAS tested by Marek, Tervo-Clemmenset al. are even worse.