Paula Rubio-Fernández

CV

I am Senior Investigator at the Multimodal Language Department at the Max Planck Institute for Psycholinguistics. I work in experimental pragmatics, employing methods from psycholinguistics and cognitive science to test pragmatic theories of communication. I am interested in the interdependence between language and social cognition, including their co-development during human maturation and their co-evolution during language change—a research program I call cultural evolutionary pragmatics.

At the Multimodal Language Department, I lead the Multimodal Reference Cluster, which investigates referential communication across modalities (including speech/sign, gaze and gesture), languages, and ages to address fundamental questions about the way human social cognition is shaped by communicative face-to-face interaction. The Cluster is divided into four groups, working on multimodal referential communication from four complementary approaches:

Reference production: How do speakers of different languages synchronize gaze, pointing, and speech in face-to-face referential communication? We address this question by comparing the use of demonstratives and other referential expressions in Turkish, Japanese, and Spanish. Our participants wear eye-tracking glasses to monitor their gaze coordination, while external cameras record their speech and pointing gestures. 

Reference comprehension: How do listeners integrate the speaker’s gaze, pointing, and speech when they interpret a referential expression? To accurately measure listener’s response (including their looking behavior via eye-tracking), we immerse our participants in a Virtual Reality where they play a referential communication task with a human-animated avatar. We are conducting the first experiment in Dutch at the MPI, but using mobile technology that will allow us to conduct the experiment in other languages during fieldwork.

Reference development: Infants break into language through pointing, and soon after start using demonstratives to establish joint attention with their caregivers. Despite the universality of these milestones, little is known about how children acquire the meaning of demonstratives across languages, and what role manual activity (including pointing gestures) play in this acquisition process. To address this question, we investigate mother-infant interaction during naturalistic toy play, focusing on the mother’s use of demonstratives and definite articles (our baseline). We use head-mounted eye-tracking to monitor gaze coordination between mother and infant, and external cameras to track their object manipulation during reference.

Reference modelling: Recent advances in multimodal language models (MLMs) have enabled systems to use text and image so naturally that users often perceive them as real conversational partners. However, existing evaluations of MLMs have largely focused on their use of vocabulary and syntax, while overlooking a fundamental class of grammatical words: indexicals. We have recently completed the first study of humans’ and MLMs’ use of indexicals in simulated face-to-face referential communication. The results confirmed the predicted difficulty hierarchy (vocabulary < possessives < demonstratives) in both groups. However, the difference between content words and indexicals was larger in MLMs, suggesting limitations in perspective-taking and spatial reasoning.

 

Curriculum Vitae

I completed a BA in Spanish Philology at the University of Oviedo (Spain), an MA in Theoretical Linguistics at the University of Reading (UK) and an MPhil in English and Applied Linguistics at the University of Cambridge, before obtaining my PhD at the University of Cambridge in 2005. 

Between 2005-2011, I was a Postdoc at the Linguistics Department at UCL funded by a British Academy Fellowship and at the Psychology Department at Princeton University funded by a Marie Curie International Fellowship. Between 2011-2014, I was a named Postdoc on a Leverhulme Research Project at UCL Psychology and Language Sciences.

In 2014, I received a Young Research Talent Grant from the Research Council of Norway and was the PI of a 4-year project on cross-linguistic pragmatics at the Philosophy Department at the University of Oslo. During that time, I also received a Mobility Grant to visit the Language Lab at the Brain and Cognitive Sciences Department at MIT. This research stay was extended to 3 years when I received funding from the Research Council of Norway for another large-scale project (2018-2023) to investigate the pragmatic development of blind, sighted and newly-sighted children in India, in collaboration with the Vision Lab at MIT and Project Prakash—a Humanitarian/Scientific organisation. During that period, I was Senior Researcher at the Philosophy Department at the University of Oslo.

In June 2023, I joined the Multimodal Language Department at the Max Planck Institute for Psycholinguistics as Senior Investigator.