Does thick description lead to smart tests? A data-based approach to rating scale construction

Abstract

Within the fields of applied linguistics and language testing, there has been a recent interest in rating scales, and how rating scales are constructed (Upshur and Turner, 1995). This is not surprising, as there is increasing concern that scores from language tests should be meaningful in applied linguistics terms. However, applied linguistics research and second language acquisition research have done little to provide descriptions of language abilities or performances which can be operationalized by language testers. Many existing descriptors for bands in rating scales are therefore barely tenable as definitions of constructs. This article looks at the definition of fluency in the literature, and proposes a qualitative and quantitative approach which may be used to produce a 'thick' description of language use, which can be used in rating scale construction. A fluency rating scale is described, and its reliability and validity assessed. The arti cle suggests that validity considerations must be addressed in the construction phase of developing rating scales, through the careful consideration of the linguis tic meaning of constructs, rather than merely as a post hoc enterprise.