Mixed Methods Research in Language Testing and Assessment

Abstract

As an alternative paradigm, mixed methods research (MMR), in general, endorses pluralism to understand the complex nature of a social world from multiple perspectives and multiple methodological lenses, each of which offers partial, yet valuable, insights. This methodological mixing is not limited to mixing of methods, but extends to the entire inquiry process. Researchers in language testing and assessment (LTA) are increasingly turning to MMR in order to understand the complexities of language acquisition and interaction among various language users, and also to expand opportunities to investigate validity claims beyond the three traditional facets of construct, content, and criterion validity. We use current conceptualizations of validity as a guiding framework to review 32 empirical MMR studies that have been published in LTA since 2007. Our systematic review encompassed multiple areas of foci, including the rationale for the use of MMR, evidence of collaboration, and synergetic effects. The analyses revealed several key trends including: (a) triangulation and complementarity were the prevalent uses of MMR in LTA; (b) the majority of the studies took place predominantly in higher education learning contexts with adult immigrant or university populations; (c) aspects of writing assessment were most frequently the focus of the studies (compared to other language modalities); (d) many of the studies explicitly addressed facets of validity, and others had significant implications for expanding notions of validity in LTA; (e) the majority of the studies avoided mixing at the data analysis stage by distinguishing data types and reporting results separately; and (f) integration occurred primarily at the discussion stage. We contend that LTA should embrace MMR through creative designs and integrative analytic strategies to seek new insights into the complexities and contexts of language testing and assessment.