Our statistical intuitions may be misleading us: Why we need robust statistics

Abstract

Most academics' intuitions about statistics follow those of naive laypeople – that is, we often think that a sample should reflect the population characteristics more closely than it does, and expect less variability in samples than is truly found in them. These intuitions may prevent us from understanding why modern developments in statistics are needed. Another intuition most researchers hold is that it is better to be conservative when performing statistics, and this may involve adjusting p-values for multiple tests, using more conservative post hoc tests, or setting an alpha value lower than .05 when possible. However, the more we try to control against making an error in being overeager to find differences, the stronger the probability that we will make an error in not finding differences that actually exist. These two forces need to be counterbalanced, and this involves increasing the power of our tests. Robust statistics can increase the power of statistical tests to find real differences. I discuss the need for robust techniques to avoid reliance on classical assumptions about the data. Examples of robust analyses with t-tests, correlation, and one-way ANOVA are shown.