Two P or Not Two P: Mendel Random Variables in Combining Fake and Genuine p-Values

Abstract

The classical tests for combining p-values use suitable statistics T(P1,…,Pn), which are based on the assumption that the observed p-values are genuine, i.e., under null hypotheses, are observations from independent and identically distributed Uniform(0,1) random variables P1,…,Pn. However, the phenomenon known as publication bias, which generally results from the publication of studies that reject null hypotheses of no effect or no difference, can tempt researchers to replicate their experiments, generally no more than once, with the aim of obtaining “better” p-values and reporting the smallest of the two observed p-values, to increase the chances of their work being published. However, when such “fake p-values” exist, they tamper with the statistic T(P1,…,Pn) because they are observations from a Beta(1,2) distribution. If present, the right model for the random variables Pk is described as a tilted Uniform distribution, also called a Mendel distribution, since it was underlying Fisher’s critique of Mendel’s work. Therefore, methods for combining genuine p-values are reviewed, and it is shown how quantiles of classical combining test statistics, allowing a small number of fake p-values, can be used to make an informed decision when jointly combining fake (from Two P) and genuine (from not Two P) p-values.