Inferential misconceptions and replication crisis

Abstract

Misinterpretations of the p value and the introduction of bias through arbitrary analytical choices have been discussed in the literature for decades. Nonetheless, they seem to have persisted in empirical research, and criticisms of p value misuses have increased in the recent past due to the non-replicability of many studies. Unfortunately, the critical concerns that have been raised in the literature are scattered over many disciplines, often linguistically confusing, and differing in their main reasons for criticisms. Misuses and misinterpretations of the p value are currently being discussed intensely under the label “replication crisis” in many academic disciplines and journals, ranging from specialized scientific journals to Nature and Science. In a drastic response to the crisis, the editors of the journal Basic and Applied Social Psychology even decided to ban the use of p values from future publications at the beginning of 2015, a fact that has certainly added fuel to the discussions in the relevant scientific forums. Finally, in early March, the American Statistical Association released a brief formal statement on p values that explicitly addresses misuses and misinterpretations. In this context, we systematize the most serious flaws related to the p value and discuss suggestions of how to prevent them and reduce the rate of false discoveries in the future.