The study of multimodality is concerned with how multiple expressive resources may be effectively combined in communication. In recent years the potential domains of applications of multimodality studies have expanded considerably. Ever more diverse combinations of expressive resources are employed in a host of media, both new and old. With this expansion it has become increasingly evident that an adequate understanding of the workings of such media will need to involve close attention to receivers’ embodied, experiential engagement with them. Nevertheless, the most widely used approaches to multimodality developed in the Saussure/Hjelmslev linguistic tradition have still not succeeded in anchoring embodiment in their theoretical frameworks in anything more than a schematic fashion. The goal of this article is to redress this balance and construct a semiotic theory of multimodality in which embodied responses are assigned an intrinsic and inalienable role. Taking the medium of narrative film as its point of departure, the article draws on the rapidly expanding body of literature on film reception addressing issues of emotional response, engagement, embodiment, and immersion to motivate an inherently embodied semiotic account that nevertheless maintains close contact with broader accounts of narrative and larger-scale “text” organization.
Multimodality and Materiality: The Interplay of Textuality and Texturality in the Aesthetics of Film
John A. Bateman is professor of applied linguistics in the Faculty of Linguistics and Literary Sciences at the University of Bremen, Germany. His main fields of research include the semiotic foundations of multimodality and the use of empirical methods for their investigation, formal ontology, and computational linguistics (particularly natural language generation, discourse, and dialogue). He has published widely in all of these areas, including Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents (2008); Multimodal Film Analysis: How Films Mean, with Karl-Heinrich Schmidt (2012); Text and Image: A Critical Introduction to the Visual/Verbal Divide (2014); and Multimodality: Foundations, Research, and Analysis; A Problem-Oriented Introduction, with Janina Wildfeuer and Tuomo Hiippala (2017).
John A. Bateman; Multimodality and Materiality: The Interplay of Textuality and Texturality in the Aesthetics of Film. Poetics Today 1 June 2019; 40 (2): 235–268. doi: https://doi.org/10.1215/03335372-7298536
Download citation file: