Is N-Hacking Ever OK? A simulation-based study

Abstract

AbstractAfter an experiment has been completed and analyzed, a trend may be observed that is “not quite significant”. Sometimes in this situation, researchers incrementally grow their sample size N in an effort to achieve statistical significance. This is especially tempting in situations when samples are very costly or time-consuming to collect, such that collecting an entirely new sample larger than N (the statistically sanctioned alternative) would be prohibitive. Such post- hoc sampling or “N-hacking” is denounced because it leads to an excess of false positive results. Here simulations are used to illustrate and explain how unplanned incremental sampling causes excess false positives. In a parameter regime representative of practice in many research fields, however, simulations show that the inflation of the false positive rate is surprisingly modest. The effect on false positive rate is only half the story. What many researchers really care about is the effect of N-hacking on the likelihood that a positive result is a real effect: the positive predictive value (PPV). This question has not been considered in the reproducibility literature. The answer depends on the effect size and the prior probability of an effect. Although in practice these values are not known, simulations show that for a wide range of values, the PPV of results obtained by N-hacking is in fact higher than that of non-incremented experiments of the same sample size and statistical power. This is because the increase in false positives is more than offset by the increase in true positives. Therefore, in many situations, adding a few samples to shore up a nearly-significant result would in fact increase reproducibility, counter to current rhetoric. To strictly control the false positive rate on the null hypothesis, the sampling plan (and all other study details) must be prespecified. But if this is not the primary concern, as in exploratory studies, collecting additional samples to resolve a borderline p value can confer previously unappreciated advantages for efficiency the positive predictive value of the generated hypotheses.