HeadTalk, HandTalk and the corpus: towards a framework for multi-modal, multi-media corpus development

Abstract

In this paper, we address a number of key methodological challenges and concerns faced by linguists in the development of a new generation of corpora: the multi-modal, multi-media corpus – that which combines video, audio and textual records of naturally occurring discourse. We contextualise these issues according to a research project which is currently developing such a corpus: the ESRC-funded Understanding New Digital Records for e-Social Science (DReSS) project based at the University of Nottingham.22 For further information, results and publications related to the project, please refer to the main DReSS website, at: http://web.mac.com/andy.crabtree/NCeSS_Digital_Records_Node/Welcome.htmlThis paper primarily explores the questions of the functionality of the corpus, identifying the problems we faced in making multi-modal corpora ‘usable’ for further research. We focus on the need for new methods for categorising and marking up multiple streams of data, using, as examples, the coding of head nods and hand gestures. We also consider the challenges faced when integrating and representing the data in a functional corpus tool, to allow for further synthesis and analysis.Here, we also underline some of the ethical challenges faced in the development of this tool, exploring the issues faced both in the collection of data and in the future distribution of video corpora to the wider research community.