Multimodality matters in numerical communication

Abstract

Modern society depends on numerical information, which must be communicated accurately and effectively. Numerical communication is accomplished in different modalities—speech, writing, sign, gesture, graphs, and in naturally occurring settings it almost always involves more than one modality at once. Yet the modalities of numerical communication are often studied in isolation. Here we argue that, to understand and improve numerical communication, we must take seriously this multimodality. We first discuss each modality on its own terms, identifying their commonalities and differences. We then argue that numerical communication is shaped critically by interactions among modalities. We boil down these interactions to four types: one modality canamplifythe message of another; it candirectattention to content from another modality (e.g., using a gesture to guide attention to a relevant aspect of a graph); it canexplainanother modality (e.g., verbally explaining the meaning of an axis in a graph); and it canreinterpreta modality (e.g., framing an upwards-oriented trend as a bad outcome). We conclude by discussing how a focus on multimodality raises entirely new research questions about numerical communication.