Statistical tests of differential susceptibility: Performance, limitations, and improvements

Abstract

AbstractStatistical tests of differential susceptibility have become standard in the empirical literature, and are routinely used to adjudicate between alternative developmental hypotheses. However, their performance and limitations have never been systematically investigated. In this paper I employ Monte Carlo simulations to explore the functioning of three commonly used tests proposed by Roisman et al. (2012). Simulations showed that critical tests of differential susceptibility require considerably larger samples than standard power calculations would suggest. The results also showed that existing criteria for differential susceptibility based on the proportion of interaction index (i.e., values between .40 and .60) are especially likely to produce false negatives and highly sensitive to assumptions about interaction symmetry. As an initial response to these problems, I propose a revised test based on a broader window of proportion of interaction index values (between .20 and .80). Additional simulations showed that the revised test outperforms existing tests of differential susceptibility, considerably improving detection with little effect on the rate of false positives. I conclude by noting the limitations of a purely statistical approach to differential susceptibility, and discussing the implications of the present results for the interpretation of published findings and the design of future studies in this area.