Chapter 4. Multimodality

Abstract

Multimodality examines how language and other resources (e.g., images and gestures) are integrated to make meaning for communication. This chapter aims to introduce two approaches to multimodality from a systemic-functional semiotic perspective, namely social semiotics and systemic functional multimodal discourse analysis (SF-MDA), and discuss their applications in applied linguistics. Specifically, this chapter begins with an overview of the theoretical underpinnings of the two approaches, focusing on their shared functionalist origins and the origin’s theoretical and methodological implications. Next, the chapter discusses the affordance of social semiotics and SF-MDA and highlights data collection procedures and analytical processes in applied linguistic studies informed by the two approaches. Finally, critiques of the two approaches are addressed with multimodal research facilitated by eye-tracking technology.