Comics Cognition: Working Memory and Situation Models in Visual Narrative Processing

Recent research in visual narrative processing, significantly informed by Visual Language Theory (Cohn, 2013), highlights the parallels between linguistic and visual modalities, particularly the role of grammar and structural principles in comprehension. While psycholinguistic approaches have advanced our understanding of visual narrative processing as akin to language, less attention has been paid to the cognitive mechanisms that facilitate this comprehension. This thesis adopts a cognitive science perspective, employing the multicomponent model of working memory and the situation model processing framework to investigate the interplay of visuospatial and verbal working memory in visual narrative processing. By addressing gaps in previous research, including the contrasting findings of Magliano et al. (2016) and Zhao et al. (2024), and testing novel stimuli designed specifically for this study, the thesis seeks to illuminate the roles of working memory and mental imagery in constructing and comprehending visual narratives. The empirical work then investigates the roles of working memory and situation model processing in visual narrative comprehension through a series of experiments using the dual task paradigm as well as established situation model metrics. Experiments 1–3 reveal that visuospatial working memory, rather than verbal working memory, underpins the processing of visual narrative sequences when individual differences in working memory are controlled. Experiment 4 confirms that visual narratives are processed within the situation model framework, evidenced by faster reading times for coherent comics compared to incoherent ones and situation model updating behaviors observed in recognition tasks. However, Experiments 5 and 6 suggest that neither nor verbal working memory directly contribute to situation model processing, indicating that working memory supports aspects of narrative comprehension but is not integral to forming or updating situation models. Finally, Experiment 7 finds that participants with better mental imagery ability exhibited longer reading times, suggesting deeper engagement with visual narratives. These findings highlight the complexity of cognitive processes in visual narrative comprehension and the distinct roles of working memory and mental imagery. This thesis makes significant contributions to understanding visual narrative processing by offering novel insights into the roles of working memory and mental imagery. The findings refine theoretical perspectives on the cognitive processes underpinning narrative comprehension, demonstrating that visuospatial working memory predominantly supports visual narrative processing, while verbal working memory plays a negligible role. Moreover, the unexpected relationship between vivid mental imagery and slower reading times suggests that mental imagery enhances narrative engagement, contributing to a deeper but more resource-intensive comprehension process. These results not only affirm the applicability of situation model frameworks to visual narratives but also advance methodological approaches for studying multimodal comprehension, providing a robust foundation for future investigations into the cognitive underpinnings of visual narrative processing.