Advancing the validity argument for standardized writing tests using quantitative rhetorical analysis

Abstract

Essay writing is a common type of constructed-response task used frequently in standardized writing assessments. However, the impromptu timed nature of the essay writing tests has drawn increasing criticism for the lack of authenticity for real-world writing in classroom and workplace settings. The goal of this paper is to contribute evidence to a validity argument for standardized writing tests. Using measurements of distances between rhetorical profiles in the corpora of interest, we examined connections between argumentative writing on standardized assessments and in external writing situations; namely, opinionated writing in academic and real-life settings. The results show that test corpora, focusing on argumentation in two standardized tests, are rhetorically similar to academic argumentative writing in a graduate-school setting, and about as similar as a corpus of civic writing in the same genre. The proximity between the test corpora and corpora representing external criteria of interest support the assessment use argument. The argumentative writing skills employed on the test are similar to the skills employed in academic and civic settings, despite the differences in the nature of the settings under which the writing samples for these different corpora are produced.