Multimodal Communicative Moves in Expositive Dialogue: Common and Novel Topic Elaboration

Abstract

The study explores the distribution and structure of multimodal clusters presenting a series of communicative moves in expositive dialogues: Request, Elaboration, and Response. We hypothesize that multimodal clustering of moves will be predetermined by the use of either common (for both participants) or novel topic elaboration as a nucleus move within the cluster. To proceed, we conduct a multimodal experiment which recorded the participants’ gesture with motion capture system (Perception Neuron Motion Capture) and gaze with eye-tracking glasses (Tobii Pro Glasses 2), as well as their speech and overall multimodal behavior with a stationary camera. The study reveals significant differences in the use of both face-oriented gaze and contactestablishing gesture as modulated by Request and Response moves within common or novel topic elaboration clusters; however, face-oriented gaze use manifests both higher frequency and diversity. Mutual face-oriented gaze prevails at the Request move preceding common topic elaboration, whereas elaborating a novel topic is found to produce a more involved gaze reaction of the listener during the Response moves. Additionally, simultaneous (by both participants) verbal move is more typical of common topic elaboration. The results evidence that social interaction and communication in expositive dialogue is processed multimodally and predetermines the role of gaze, gesture and verbal moves in communicative moves clusters.