An Updated Recommendation for Multiple Comparisons

Abstract

Instructors of introductory and intermediate statistics courses often teach the use of analysis of variance (ANOVA) for the purpose of comparing more than two group means and pairwise comparison procedures (PCPs) to determine which group means differ from one another following a statistically significant ANOVA test. SPSS provides 18 PCPs. The purpose of this study was to determine which PCP has the best power and maintains Type I error control both when assumptions of equal sample size and equal variance are met and when they are violated, so as to provide a single resource for use in introductory and intermediate applied statistics courses. Testing the PCPs with simulated data revealed that only the four tests developed to be used under assumption violations adequately controlled Type I error, so we recommend using one of these procedures. Power results were similar for all four of these tests, but were slightly higher for the Games-Howell test than for the others.