The Challenge of Assessing Content and Coherence

Abstract

Content and coherence are the categories most difficult to evaluate fairly when raters use analytic scoring scales. Readers inevitably interpret texts in their own idiosyncratic ways, depending on their knowledge, experience, ethical considerations, and other personal biases that they cannot completely set aside when grading a text. This is also true for descriptors, which are themselves short texts. To make matters worse, due to the very nature of writing but also the lack of consensus among experts in discourse research, writing theory, and writing assessment, descriptors are categorized vaguely and inconsistently. As a result, raters seeking useful evaluation criteria are confronted with descriptors that cover the same concept, such as “relevance”, being categorized in one set of criteria as relating to the content of the written text and in another as belonging to the category of coherence. Nevertheless, the objectivity of the evaluation of written work can be increased. The article examines the relationship between content and coherence, which is reflected in the way the two concepts are defined in the relevant literature, as well as in some descriptors used in two grading scales used in Slovenia. The empirical part of the paper presents a case study involving 46 secondary school teachers, whose responses to a questionnaire confirm the subjectivity of the understanding of individual descriptors and the need for adequate training of teachers in the use of analytic scoring scales, regular standardization in the schools where they work, evaluation of the assessment scales they use and their possible adaptation.