Univariate Tests for Phase Capacity: Tools for Identifying When to Modify a Survey’s Data Collection Protocol

Abstract

Abstract To mitigate the potentially harmful effects of nonresponse, most surveys repeatedly follow up with nonrespondents, often targeting a response rate or predetermined number of completes. Each additional recruitment attempt generally brings in a new wave of data, but returns gradually diminish over the course of a fixed data collection protocol, as each subsequent wave tends to consist of fewer responses than the last. Consequently, point estimates begin to stabilize. This is the notion of phase capacity, suggesting some form of design change is in order, such as switching modes, increasing the incentive, or, as is considered exclusively in this research, discontinuing the nonrespondent follow-up campaign altogether. A previously proposed test for phase capacity calls for multiply imputing nonrespondents’ missing data to assess, retrospectively, whether the most recent wave of data significantly altered a key, nonresponse-adjusted point estimate. This study introduces a more flexible adaptation amenable to surveys that instead reweight the observed data to compensate for nonresponse. Results from a simulation study and application indicate that, all else equal, the weighting version of the test is more sensitive to point estimate changes, thereby dictating more follow-up attempts are warranted.