You are here: Home Departments Neurobiology of Language Linking language to the visual world

Neurobiology of Language -

Linking language to the visual world

A fundamental aspect of human communication is that it allows us to refer to the things in the world around us. Indeed we often talk about the things in our immediate environment. Our hands help us do so, by pointing out the exact object, person, or event we are talking about. It has long been unclear how the brain integrates auditory (speech) and visual (pointing gestures, object) information in very common everyday situations in which a speaker refers to an object by using speech and a pointing gesture. A recently published fMRI study by David Peeters and colleagues sheds light on this important issue.
Linking language to the visual world

subharnab voor Flickr via CC BY 2.0

While in an MRI scanner, participants were presented with pictures of a speaker and two objects while they concurrently listened to her speech. In each picture, one of the objects was singled out, either through the speaker's index-finger pointing gesture or through a visual cue that made the object perceptually more salient in the absence of gesture. A mismatch (compared to a match) between speech and the object singled out by the speaker's pointing gesture led to enhanced activation in left inferior frontal gyrus and bilateral posterior middle temporal gyrus, showing the importance of these areas in matching between speech and object-referent. Moreover, a match (compared to a mismatch) between speech and the object made salient through a visual cue led to enhanced activation in the mentalizing system, arguably reflecting an attempt to converge on a jointly attended referent in the absence of pointing.

These novel findings for the first time shed light on the brain areas involved in the core communicative process of comprehending a speaker's multimodal referential act and stress the power of pointing as an important natural device to link speech to objects. Future work will continue to investigate the cognitive and neural mechanisms involved in understanding pointing gestures by using virtual reality technology.


Peeters, D., Snijders, T. M., Hagoort, P., & Ozyurek, A. (2017). Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues. Neuropsychologia, 95, 21-29.


Would you like to read more about pointing gestures? Kennislink recently published an article (in Dutch) on how pointing gestures played an important role in language evolution.

Would you like to read more about virtual reality? Kennislink published another article on virtual reality research at the MPI.

Neurobiology of Language

What is the neurobiological infrastructure for the uniquely human capacity for language? The focus of the Neurobiology of Language Department is on the study of language production, language comprehension, and language acquisition from a cognitive neuroscience perspective. Read more...

Director: Peter Hagoort

Secretary: Carolin Lorenz


Flag NL Het talige brein