A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling

Abstract

Despite that common method variance (CMV) is widely regarded as a serious threat to the validity of findings based on self-reports, there is insufficient research on its confounding influence. We extend Evans’s (1985) pioneering work, and the more recent works by Ostroff, Kinicki, and Clark (2002) and Siemsen, Roth, and Oliveira (2010), to delineate the influence of CMV in a two-level hierarchical linear model based on self-report data. Our simulation results clearly show that in the absence of true effects, it is extremely unlikely for CMV to generate significant cross-level interactions. In fact, if a true cross-level interaction exists, CMV tends to lower the likelihood of its identification and erroneously underestimate the regression coefficient. Our simulation results also show that CMV may lead to a false significant cross-level main effect and overestimate the regression coefficient when no true effect exists. To reduce the probability of Type I errors, we show that raising the significance level to .01, the split sample strategy, and the addition of more CMV contaminated variables are effective in the vast majority of real-life situations and are more effective than increasing the number of groups or persons in each group. Both the split sample strategy and the addition of more CMV contaminated variables are also effective in reducing parameter bias when no true cross-level main effect exists. Trade-offs associated with different strategies are discussed.