Quality and quantity in translation evaluation: A starting point

Abstract

In academic environments, the question of quality is often related to that of quantity. This paper presents a quantitative analysis of translation evaluation quality by focusing on several issues: a) the number of actions performed by evaluators as related to quality judgments; b) evaluator coincidence in mistakes detected (saliency of phenomena) as related to their nature; and c) quality judgment as related to the presence or absence of the most salient phenomena in the translated texts. Four groups of participants evaluated 48 translations; the results discussed derive from their actual performance. The results suggest that evaluators exhibiting a minimum level of detail in their work should be thorough in their evaluative work for, if they are not, they run the risk of being too generous in their evaluations. Among the error judgments for which there is the highest degree of coincidence are those on divergent interpretation of the original text, terminology, syntax, (language) use and misprints. Conversely, evaluators do not usually coincide in pointing out punctuation, format, proper nouns, clarity, cohesion or appropriateness phenomena. Finally, texts including the most salient phenomena receive worse grades than those without them; this points to an implicit hierarchy of phenomena according to the extent to which evaluators coincide in pointing them out.