Adjusting Bilingual Ratings by Retest Reliability Improves Estimation of Translation Quality

Abstract

The quality of cross-language scale translations is often explored by having bilingual participants complete the scale in both languages and then correlating their scores. However, low cross-language correlations can be observed due to score unreliability rather than due to poor scale translation. McCrae, Yik, Trapnell, Bond, and Paulhus suggested that a better indicator of translation quality can be formed by dividing the raw cross-language correlation by the same-language retest correlations over a similar measurement interval. Here, we illustrate how this method can be extended to evaluate the translation quality of individual items. We translated the English version of the Inventory of Individual Differences in the Lexicon (IIDL) into Chinese, and within a single survey session participants either completed the instrument either in both languages ( N = 151 bilingual participants) or twice in Chinese ( N = 94) or in English ( N = 82). Finally, additional bilingual participants ( N = 46) rated the perceived translation quality of each item. Variation in the cross-language correlations across items predicted perceived translation quality, however, adjusting for same-language retest correlations resulted in significantly stronger indicators of perceived translation quality. The present study thus indicates the validity of McCrae et al.’s general method, and demonstrates that it can be extended to designs where all participants complete a single test session and can be applied to evaluate the quality of translations of single items.