Event Construal in Text and Image. Gaze Behavior and Discourse Interpretations

Abstract

The study explores the combinatorial prevalence effect in Event construal techniques in text and image components of heterosemiotic book pages. We hypothesize that their activity and contingency affect their interpretation, here tested in the oculographic experiment and discourse responses check. To proceed, we develop the parametric system applied for 100 book pages annotation and further statistical analysis. This study reveals the relevance of Truth, Type, Relation, Manageability, Completeness, Instantness, Achievement, Evaluation, Space location, Time location, Repeatability, Cause and effect parameter groups in Event construal in text and image as well as their resonance in concomitant activity. To select the samples serving as stimuli in the oculographic experiment, we apply Principal component analysis, which assigns Uniqueness indices to the samples, here ranging from 0.111 to 0.675, and provides diversity of Event construal techniques to be tested in terms of their interpretation. The results evidence that participants applied different text and image attention distribution patterns with longer fixations on text component in case the image displayed physical contact, static and desirable events. When the creation or destruction events, events-achievement, events located in time or causal events were not present in the text, the participants were more likely to address the image, not the text. Parameter activity also affects the choice of Descriptive, Narrative and Speculative discourse responses, with a restricted number of parameters stimulating Narrative discourse, with a restricted in text and vast in image number of parameters stimulating Speculative discourse, which evidences in favor of their more predetermined and predicted character. Hopefully, the results may be used to predict the interpretation effects and to further cognitive linguistic and semiotic research coordination.