A Permutation Test on Complex Sample Data

Abstract

Abstract Permutation tests are a distribution-free way of performing hypothesis tests. These tests rely on the condition that the observed data are exchangeable among the groups being tested under the null hypothesis. This assumption is easily satisfied for data obtained from a simple random sample or a controlled study after simple adjustments to the data, but there is no general method for adjusting survey data collected using a complex sample design to allow for permutation tests. In this article, we propose a general method for performing a pseudo-permutation test that accounts for the complex sample design. The proposed method is not a true permutation test in that the new values do not come from the set of observed values in general but of an expanded set of values satisfying a random-effects model on the clustered residuals of a design-consistent estimating equation. We provide a set of conditions under which this procedure leads to consistent test results. Tests using a simulated population and an application analyzing US Bureau of Labor Statistics consumer expenditure data comparing the performance of the proposed method to permutation tests that ignore the sample design demonstrate that it is necessary to account for the design features in order to obtain reasonable p value estimates.