Prompt effects on assessment of interactional competence in a group oral test

Abstract

AbstractThis study serves as an attempt to validate an in‐house oral proficiency test (hereafter referred to as OPT) with a particular focus on prompt effects on assessment of interactional competence, an issue that has attracted burgeoning interest but not been adequately investigated yet. Prompts employed in the parallel versions of the group discussion task, which were supposed to be comparable, were found to differ in two dimensions, that is, topical domain and syntactic structure. The transcribed interactions from 136 sophomore test takers (34 groups) sitting in the first administration of the OPT were coded through a rigorous procedure and examined using conversation analysis. Five patterns of interaction were identified with distinct features based on two dimensions, mutuality and equality. Marked differences in distribution of patterns of interaction were detected in discourse elicited by different prompts, which should be mainly attributed to the syntactic structure of the prompts. The open‐question prompts were found more likely to elicit “conversation‐like” interaction with higher mutuality, characterized by the collaborative and unbalanced‐collaborative patterns, while the closed‐question prompts tended to elicit more “solo versus solo” interaction with lower mutuality, featured by the parallel and unbalanced‐parallel patterns. The influence of topic domain, however, was minimal. The results thereby constitute rebuttals that weaken meaningfulness and impartiality of the claim of interpretations about the construct within Bachman and Palmer's assessment use argument framework and suggest close attention from the test developers. The implications for prompt design, and development and validation of the group oral test are further provided.