Talk to the hand: How gesture makes spoken conversation smoother

What was the main question in your dissertation?

In everyday conversation, people take turns speaking: you say something, then I say something, and so on. Typically, these exchanges happen rapidly, with just a short pause or a short overlap between speaking turns. When you are engaged in a conversation with someone, this fast back-and-forth feels easy, but the underlying processes in your mind are not simple at all. Understanding how we manage to respond quickly in conversations is one of the big questions in the field of psycholinguistics. 

Adding to the complexity, most conversations happen in a face-to-face setting, in which language is multimodal: this means that we do not only communicate with speech, but also with bodily signals, such as hand gestures. In my PhD, I investigated whether seeing such hand gestures could actually help us to respond so quickly during conversation. 

 

Can you explain the background a bit more?

The idea that seeing gestures may help us respond quickly in a conversation can be illustrated by this example. Imagine that you are at a party and someone asks you “Would you like to...”. Based on just these words, you can make some guesses about how the question will continue (e.g., “dance” or “go home”), but you don’t yet have enough information to start preparing a response. However, if you saw that the speaker made a gesture as if they were drinking from a glass, you could perhaps already predict that your friend will ask whether you would like to drink something. Therefore, you could already start planning your reply (“Yes, please”) and respond rapidly to your friend’s question.

 

Why is it important to answer this question?

Face-to-face conversation is the setting where language has evolved, where children acquire it, and where it is most often used. Therefore, in order to truly understand how language works, we should study how language is used in this setting. This means that we should take into account the fact that language is more than just speech (and also includes visual signals such as hand gestures) as well as the turn-taking that happens during conversations. 

 

Can you tell us more about one particular project that you enjoyed?

My favourite project was the third chapter of my PhD. In my first two chapters, I showed that seeing gestures facilitates fast responding during conversation, and that gestures (e.g., the drinking gesture) typically start before their corresponding speech (e.g., the word “drink”). I was curious whether this early timing of gestures would allow listeners to use the gestures to predict upcoming words. 

For this study, I used virtual avatars that asked questions, with or without a corresponding gesture. We took gestures that people had used in actual conversations, and recreated them in the avatar. This was quite a lot of work! I also had to learn EEG for this project, which was challenging at first, but very useful in the end to develop new skills. There were also some big technical issues along the way, so when the results finally came in, after all this work and all these challenges, I was very happy!  

What inspired you to choose your research topic?

During my master’s, I took Professor Aslı Özyürek’s course on hand gestures and I was immediately hooked: I felt like this was the first course where we really learned about how language is used in daily life. During my PhD, I learned even more about how language is used in its most common setting: face-to-face conversation. It really motivates me when a research topic is concrete and connected to everyday life, and my topic was just like that: In everyday life, there are always people around somewhere having a conversation and using their hands to gesture! 

What was the most rewarding or memorable moment during your PhD journey?

When I started my PhD, I was really scared of having to defend my dissertation in public all the way at the end: I just couldn’t imagine myself standing there in the Aula, with all these people watching, answering difficult questions without panicking. One of my motivations for starting the PhD was actually that I wanted to become someone who could do that: a smarter, more confident and less anxious version of myself. I feel very happy to have achieved that goal, I actually really enjoyed the defense and I felt very proud of myself the whole day and the weeks after. 

What do you want to do next?

I am currently working as a postdoctoral researcher in Prof. Judith Holler’s lab at the Donders Institute in Nijmegen. In my new project, I investigate how people communicate their intentions during face-to-face conversation. For example, how do people signal that their question “How do you think it’s going?” is a sarcastic joke rather than an honest request for someone’s opinion? How are these intentions communicated with what people say, their intonation, and their bodily signals (e.g., facial expressions)? And how do listeners pick up on these cues and decide what the appropriate response is to the question? I’m really excited that I get to study this for the next years!