Abstract
AbstractCommunication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.
Funder
ZonMw
H2020 European Research Council
Nederlandse Organisatie voor Wetenschappelijk Onderzoek
Publisher
Springer Science and Business Media LLC
Subject
Computer Networks and Communications,Human-Computer Interaction,Information Systems,Software
Reference53 articles.
1. Baker, A., van den Bogaerde, B., Pfau, R., et al.: The Linguistics of Sign Languages: An Introduction. John Benjamins (2016). https://doi.org/10.1075/z.199
2. Barr, D.J., Levy, R., Scheepers, C., et al.: Random effects structure for confirmatory hypothesis testing: keep it maximal. J. Mem. Lang. 68(3), 255–278 (2013). https://doi.org/10.1016/j.jml.2012.11.001
3. Bates, D., Maechler, M., Bolker, B., et al.: lme4: Linear mixed-effects models using Eigen and S4. http://CRAN.R-project.org/package=lme4, r package version 1.1 (2014)
4. Coerts, J.: Nonmanual grammatical markers: an analysis of interrogatives, negations and topicalisations in Sign Language of the Netherlands. PhD thesis, University of Amsterdam (1992)
5. Courty, N., Gibet, S.: Why is the creation of a virtual signer challenging computer animation? In: Motion in Games: Third International Conference (MIG 2010), pp. 290–300. Springer, Berlin (2010). https://doi.org/10.1007/978-3-642-16958-8_27