Publications

Abstract

Dynamic time warping (DTW) is a well-known algorithm used to assess the similarity between signals of varying lengths. Initially developed for automatic speech recognition, DTW has found applications in psycholinguistics, particularly in analyzing gesture form similarity. An open question in this domain is how effectively DTW captures gesture form similarity. Here, we validate DTW against human annotations of gesture form similarity across two multimodal interaction corpora and explore its utility as an automatic, continuous measure of gesture form similarity. Our findings reveal weak to moderate correlations between DTW distance and the number of similar gesture features – such as handshape, movement, orientation, and position – suggesting that DTW serves as a useful proxy for gesture form similarity. Additionally, we highlight the importance of qualitative analysis of raw data and DTW predictions in enhancing DTW’s predictive accuracy. Our study offers a rigorous validation of DTW as a measure of gesture form similarity and presents a detailed framework for preprocessing motion tracking data and calculating DTW distance. While none of the methods is perfect, the combination of automatic and manual measures provides a comprehensive approach to understanding and measuring gesture form similarity.

Abstract

Co-speech gestures emerge from the interaction between visuospatial experience and speech formulation, but what happens when visual experience is absent from birth? Blind individuals gesture in ways consistent with their native language, albeit they gesture less frequently. However, nothing is known about whether their gesture form differs from that of sighted individuals. Using computer vision technologies, this study examines whether gesture kinematics are shaped by visuospatial experience. In an auditory motion-description task, we analyzed spontaneous path gestures—depicting motion trajectories—produced by 20 congenitally blind, 21 blindfolded, and 21 sighted Turkish speakers. We assessed the gestures’ alignment with actual motion paths, along with duration, size, and speed. Blind participants produced gestures that were more precisely aligned with the motion trajectories, as well as larger and longer. These findings show that altered spatial cognition in blindness enhances gesture precision. The study underscores the role of sensorimotor experience in shaping gesture form.

Abstract

When we communicate with others, we often repeat aspects of each other's communicative behavior such as sentence structures and words. Such behavioral alignment has been mostly studied for speech or text. Yet, language use is mostly multimodal, flexibly using speech and gestures to convey messages. Here, we explore the use of alignment in speech (words) and co-speech gestures (iconic gestures) in a referential communication task aimed at finding labels for novel objects in interaction. In particular, we investigate how people flexibly use lexical and gestural alignment to create shared labels for novel objects and whether alignment in speech and gesture are related over time. The present study shows that interlocutors establish shared labels multimodally, and alignment in words and iconic gestures are used throughout the interaction. We also show that the amount of lexical alignment positively associates with the amount of gestural alignment over time, suggesting a close relationship between alignment in the vocal and manual modalities.