Systematic effects in the rating of second-language speaking ability: test method and learner discourse

Abstract

Major differences exist in two approaches to the study of second-language performance. Second-language-acquisition (SLA) research examines effects upon discourse, and is typically unconcerned with scores. Language-testing (LT) research investigates effects upon scores, generally without reference to discourse. Within a general framework of test taking and scoring, we report research from these two fields as it relates to questions of systematic effects on second-language tests. We then examine findings incidental to a test-development project. The findings were consistent with LT research into systematic effects of task and rater on ratings, and with SLA research into systematic effects of task on discourse. Using empirically derived scales as indicators of salient features of discourse, we infer that task type influences strategies for assessing language performance. Explanations for these joint findings are not afforded by either standard LT or SLA perspectives. There is no theory of method to explain how particular aspects of method affect discourse, how those discourse differences are then reflected in ratings and how task features influence the basis for judgement. We conclude that a full account of performance testing requires a paradigm that incorporates relationships that are not specified in either the major language-testing research tradition or the tradition of second-language-acquisition research.