Using cluster-based permutation tests to estimate MEG/EEG onsets: how bad is it?

Abstract

AbstractLocalising effects in space, time and other dimensions is a fundamental goal of magneto- and electro-encephalography research. A popular exploratory approach applies mass-univariate statistics that are corrected for multiple comparisons using cluster inferences, by pooling together neighbouring effects. However, these cluster-based methods have an important limitation: because the inference is at the cluster level, there is no explicit localisation, and one must be cautious about interpreting when and where effects start and end. Sassenhagen & Draschkow (2019) provided an important reminder of this limitation. They also reported results from a simulation, suggesting that estimated onsets are both positively biased and underestimate real onsets in a large proportion of simulated experiments. However, the simulation did not provide results from other methods as a comparison. Using a new simulation, here I demonstrate that cluster-based inferences perform well relative to other standard methods that providepvalues at every testing point: correcting for multiple comparisons using the false discovery rate adjustment leads to higher underestimation, while a maximum statistic leads to higher overestimation.Researchers using cluster-based inferences will report more accurate onset estimates in the long run than if they use these alternative methods. It is also worth keeping in mind that all common approaches to infer effect onsets rely on a statistical fallacy, because they never explicitly test the implied interaction between statistically significant and non-significant points. Better approaches to onset estimation can be implemented, for instance using change point detection algorithms.