Esam Ghaleb Presents at Joint Special Interest Group: NLP@Utrecht

On 25 September 2025, Esam Ghaleb presented his research at the 2025–2026 Period-1 Meeting of NLP@U, hosted by the Faculty of Humanities at Utrecht University.

In his talk, “Speech and Gesture Interaction in Face-to-Face Dialogue: From Analysis to Generation,” Esam shared recent research on modelling speech and gestures in face-to-face interactions — covering both the analysis of natural communication and the generation of gestures accompanying speech.

Key highlights included:

• Multimodal gesture representations – showing that self-supervised pre-training produces stronger gesture representations when grounded in co-occurring speech. These representations align with human judgments of gesture similarity in dialogue and improve object reference prediction, even without speech input at interface.

• Speech-driven gesture generation – introducing a model that integrates semantic, acoustic, and prosodic cues to produce semantically coherent, speech-aligned gestures. The model produces gestures that better match the content and timing of what is said, improving gestural realization in virtual avatars.

Together, these studies point toward practical directions towards multimodal gesture-aware conversational systems that not only analyze human gesturing but also generate context-appropriate gestures — advancing naturalistic human-computer interaction.

About NLP@Utrecht

The Joint Special Interest Group: NLP@Utrecht (NLP@U) is a collaborative forum that brings together researchers and students from Utrecht University and beyond who are interested in natural language processing. The group meets regularly to exchange ideas, present ongoing work, and foster interdisciplinary collaboration in the field of language, communication, and technology.