Creating subject competence translation tests with GPT-4: A case study in English-to-Turkish translations in the engineering domain

Abstract

Abstract As Machine Translation (MT) technologies become more advanced, the translation errors they generate are often increasingly subtle. When MT is integrated in ‘Human-in-the-Loop’ (HITL) translation workflows for specialized domains, successful Post-Editing (PE) hinges on the humans involved having in-depth subject competence, as knowledge of the specific terminology and conventions are essential to produce accurate translations. One way of assessing an individual’s expertise is through manual translation tests, a method traditionally used by Language Service Providers (LSPs) and translator educators alike. While manual evaluation can provide the most comprehensive overview of a translator’s abilities, they have the disadvantage of being time-consuming and costly, especially when large numbers of subjects and language pairs are involved. In this work, we report on the experience of creating automated tests with GPT-4 for subject competence assessment in the translation of English-to-Turkish engineering texts in HITL translation workflows. While there may be a level of usefulness in the resulting tests, they are not fit for direct implementation without further refinement.