Correlations in Social Neuroscience Aren't Voodoo: Commentary on Vul et al. (2009)

Abstract

Vul, Harris, Winkielman, and Pashler (2009) , (this issue) claim that many brain—personality correlations in fMRI studies are “likely … spurious” (p. 274), and “should not be believed” (p. 285). Several of their conclusions are incorrect. First, they incorrectly claim that whole-brain regressions use an invalid and “nonindependent” two-step inferential procedure, a determination based on a survey sent to researchers that only included nondiagnostic questions about the descriptive process of plotting one's data. We explain how whole-brain regressions are a valid single-step method of identifying brain regions that have reliable correlations with individual difference measures. Second, they claim that large correlations from whole-brain regression analyses may be the result of noise alone. We provide a simulation to demonstrate that typical fMRI sample sizes will only rarely produce large correlations in the absence of any true effect. Third, they claim that the reported correlations are inflated to the point of being “implausibly high.” Though biased post hoc correlation estimates are a well-known consequence of conducting multiple tests, Vul et al. make inaccurate assumptions when estimating the theoretical ceiling of such correlations. Moreover, their own “meta-analysis suggests that the magnitude of the bias is approximately .12—a rather modest bias.