Semantically Congruent Visual Stimuli Can Improve Auditory Memory

Abstract

We investigated the effects of audiovisual semantic congruency on recognition memory performance. It has been shown previously that memory performance is better for semantically congruent stimuli that are presented together in different modalities (e.g., a dog’s bark with a picture of the dog) during encoding, compared to stimuli presented together with an incongruent or non-semantic stimulus across modalities. We wanted to clarify whether this congruency effect is also present when the effects of response bias and uncertainty of stimulus type are removed. The participants memorized auditory or visual stimuli (sounds, spoken words or written words), which were either presented with a semantically congruent stimulus in the other modality or presented alone during encoding. This experimental paradigm allowed us to utilize signal detection theory in performance analysis. In addition, it enabled us to eliminate possible effects caused by intermingling congruent stimuli with incongruent or non-semantic stimuli, as previously done in other studies. The memory of sounds was facilitated when accompanied by semantically congruent pictures or written words, in comparison to sounds presented in isolation. The memory of spoken words was facilitated by semantically congruent pictures. However, written words did not facilitate memory of spoken words, or vice versa. These results suggest that semantically congruent verbal and non-verbal visual stimuli presented in tandem with auditory counterparts, can enhance the precision of auditory encoding, except when the stimuli in each modality are both verbal.