A logical analysis of null hypothesis significance testing using popular terminology

Abstract

Abstract Background Null Hypothesis Significance Testing (NHST) has been well criticised over the years yet remains a pillar of statistical inference. Although NHST is well described in terms of statistical models, most textbooks for non-statisticians present the null and alternative hypotheses (H0 and HA, respectively) in terms of differences between groups such as (μ1 = μ2) and (μ1 ≠ μ2) and HA is often stated to be the research hypothesis. Here we use propositional calculus to analyse the internal logic of NHST when couched in this popular terminology. The testable H0 is determined by analysing the scope and limits of the P-value and the test statistic’s probability distribution curve. Results We propose a minimum axiom set NHST in which it is taken as axiomatic that H0 is rejected if P-value< α. Using the common scenario of the comparison of the means of two sample groups as an example, the testable H0 is {(μ1 = μ2) and [($$\overline{x}$$ x ¯ 1 ≠ $$\overline{x}$$ x ¯ 2) due to chance alone]}. The H0 and HA pair should be exhaustive to avoid false dichotomies. This entails that HA is ¬{(μ1 = μ2) and [($$\overline{x}$$ x ¯ 1 ≠ $$\overline{x}$$ x ¯ 2) due to chance alone]}, rather than the research hypothesis (HT). To see the relationship between HA and HT, HA can be rewritten as the disjunction HA: ({(μ1 = μ2) ∧ [($$\overline{x}$$ x ¯ 1 ≠ $$\overline{x}$$ x ¯ 2) not due to chance alone]} ∨ {(μ1 ≠ μ2) ∧ [$$(\overline{x}$$ ( x ¯ 1 ≠ $$\overline{x}$$ x ¯ 2) not due to (μ1 ≠ μ2) alone]} ∨ {(μ1 ≠ μ2) ∧ [($$\overline{\boldsymbol{x}}$$ x ¯ 1≠$$\overline{\boldsymbol{x}}$$ x ¯ 2) due to (μ1 ≠ μ2) alone]}). This reveals that HT (the last disjunct in bold) is just one possibility within HA. It is only by adding premises to NHST that HT or other conclusions can be reached. Conclusions Using this popular terminology for NHST, analysis shows that the definitions of H0 and HA differ from those found in textbooks. In this framework, achieving a statistically significant result only justifies the broad conclusion that the results are not due to chance alone, not that the research hypothesis is true. More transparency is needed concerning the premises added to NHST to rig particular conclusions such as HT. There are also ramifications for the interpretation of Type I and II errors, as well as power, which do not specifically refer to HT as claimed by texts.