The Burden of Reliability: How Measurement Noise Limits Brain-Behaviour Predictions

Abstract

AbstractMajor efforts in human neuroimaging strive to understand individual differences and find biomarkers for clinical applications by predicting behavioural phenotypes from brain imaging data. An essential prerequisite for identifying generalizable and replicable brain-behaviour prediction models is sufficient measurement reliability. However, the selection of prediction targets is predominantly guided by scientific interest or data availability rather than reliability considerations. Here we demonstrate the impact of low phenotypic reliability on out-of-sample prediction performance. Using simulated and empirical data from the Human Connectome Projects, we found that reliability levels common across many phenotypes can markedly limit the ability to link brain and behaviour. Next, using 5000 subjects from the UK Biobank, we show that only highly reliable data can fully benefit from increasing sample sizes from hundreds to thousands of participants. Overall, our findings highlight the importance of measurement reliability for identifying brain–behaviour associations from individual differences.