Embodied Conversational Agents in eHealth: How Facial and Textual Expressions of Positive and Neutral Emotions Influence Perceptions of Mutual Understanding

Abstract

Abstract Embodied conversational agents (ECAs) could engage users in eHealth by building mutual understanding (i.e. rapport) via emotional expressions. We compared an ECA’s emotions expressed in text with an ECA’s emotions in facial expressions on users’ perceptions of rapport. We used a $2 \times 2$ design, combining a happy or neutral facial expression with a happy or neutral textual expression. Sixty-three participants (mean, 48$ \pm $22 years) had a dialogue with an ECA on healthy living and rated multiple rapport items. Results show that participants’ perceived rapport for an ECA with a happy facial expression and neutral textual expression and an ECA with a neutral facial expression and happy textual expression was significantly higher than the neutral value of the rapport scale ($P = 0.049$ and $P = 0.008$, respectively). Furthermore, results show no significant difference in overall rapport between the conditions ($P = 0.062$), but a happy textual expression for an ECA with a neutral facial expression shows higher ratings of the individual rapport items helpfulness ($P = 0.019$) and enjoyableness ($P = 0.028$). Future research should investigate users’ rapport towards an ECA with different emotions in long-term interaction and how a user’s age and personality and an ECA’s animations affect rapport building. Optimizing rapport building between a user and an ECA could contribute to achieving long-term interaction with eHealth.