Abstract

Speech is inherently variable, requiring listeners to apply adaptation mechanisms to deal with the variability. A proposed perceptual adaptation mechanism is recalibration, whereby listeners learn to adjust cognitive representations of speech sounds based on disambiguating contextual information. Most studies on the role of recalibration in speech perception have focused on variability in particular speech segments (e.g., consonants/vowels), and speech has mostly been studied in isolation. However, speech is often accompanied by visual bodily signals like hand gestures and is thus multimodal. Moreover, variability in speech extends beyond segmental aspects alone and also affects prosodic aspects, like lexical stress. We currently do not understand well how listeners adjust their representations of lexical stress patterns to different speakers. In three experiments, we investigated recalibration of lexical stress perception, driven by lexico-orthographical information (Experiment 1) and by manual beat gestures (Experiments 2-3). Across experiments, we observed that these types of disambiguating information (presented during an initial brief exposure phase) lead listeners to adjust their representations of lexical stress, with lasting consequences for subsequent spoken word recognition (in an audio-only test phase). However, evidence for generalization of this recalibration to segmentally different words was mixed as it was found only in the final experiment. These results highlight that recalibration is a plausible mechanism for suprasegmental speech adaption in everyday communication and show that even the timing of simple hand gestures can have a lasting effect on auditory speech perception.

Abstract

Human communication is inherently multimodal. Auditory speech but also visual cues can be used to understand another talker. This is especially known about the perception of segments of speech (i.e., speech sounds). However, less is known about the influence of visual information on the perception of suprasegmental aspects of speech like lexical stress. This study investigated the influence of different visual information (e.g., facial cues & beat gestures) on the perception of lexical stress and found that beat gestures, but not facial cues affect lexical stress perception. These results highlight the importance of considering suprasegmental aspects of language in multimodal contexts and expand our understanding of audiovisual speech perception and integration.