Evaluation of American Sign Language Generation by Native ASL Signers

Abstract

There are many important factors in the design of evaluation studies for systems that generate animations of American Sign Language (ASL) sentences, and techniques for evaluating natural language generation of written texts are not easily adapted to ASL. When conducting user-based evaluations, several cultural and linguistic characteristics of members of the American Deaf community must be taken into account so as to ensure the accuracy of evaluations involving these users. This article describes an implementation and user-based evaluation (by native ASL signers) of a prototype ASL natural language generation system that produces sentences containing classifier predicates, which are frequent and complex spatial phenomena that previous ASL generators have not produced. Native signers preferred the system's output to Signed English animations -- scoring it higher in grammaticality, understandability, and naturalness of movement. They were also more successful at a comprehension task after viewing the system's classifier predicate animations.