A General Experimentwise Error Rate for Multiple Significance Tests

Abstract

No current method of controlling error rate is appropriate for all experiments. When the error rate is set at traditional levels a per comparison error rate can yield too high a proportion of Type I errors, while an experimentwise error rate can be too conservative because the purpose of the experiment is not taken into account. A definition of error rate is proposed in which the number of significant outcomes needed to answer the question of interest is considered and a distinction is made between tests of fundamental importance and those of only subsidiary interest. The definition provides a systematic method of unequally allotting the error rate such that more power is provided for tests of crucial interest and for experiments in which several significant results are required.